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FOREWORD 


The  use  of  simulations  in  U.S.  Army  training  continues  to  increase,  as  does  the  need  for 
tools  and  techniques  for  exploiting  simulation  capabilities.  The  U.S.  Army  Research  Institute 
has  led  the  development  of  structured  training  approaches  providing  such  tools  and  techniques, 
primarily  through  work  accomplished  in  the  Armored  Forces  Research  Unit  (AFRU)  at  Fort 
Knox,  Kentucky.  Experience  with  structured  simulation-based  training  has  led  to  the  recognition 
of  a  need  to  provide  a  comprehensive  system  to  “train  the  trainer.”  The  Close  Combat  Tactical 
Trainer  (CCTT)  magnifies  this  need  since  an  experienced  full-time  training  team  is  not  provided 
to  conduct  training.  Commanders  and  other  trainers  need  to  understand  the  capabilities  of  the 
CCTT,  and  be  able  to  customize  structured  training  to  maximize  the  benefit  of  the  CCTT  in  their 
unit  training  strategy. 

This  report  describes  the  design,  prototype  development,  and  formative  evaluation  of 
computer  software  that  assists  commanders  or  other  unit  trainers  to  develop  and  manage 
structured  CCTT  training.  This  effort  was  entitled  "Commanders  Integrated  Training  Tool 
(CITT)  for  the  Close  Combat  Tactical  Trainer."  The  AFRU  accomplished  this  effort  as  part  of 
Work  Package  2124,  "Strategies  for  Training  and  Assessing  Armor  Commanders’  Performance 
with  Devices  and  Simulations  (STRONGARM)."  The  relevant  requirements  document  is  a 
Memorandum  for  Record  between  the  AFRU  and  the  Project  Manager  for  the  Combined  Arms 
Tactical  Trainer  (PM  CATT),  entitled  "Structured  Training  for  the  Close  Combat  Tactical 
Trainer,"  dated  25  July  1997. 

The  CITT  software  design  and  prototype  have  been  provided  to  the  US  Army  Training 
Support  Center,  and  to  the  PM  and  the  Training  and  Doctrine  Command  System  Manager  for 
CATT.  The  CCTT  instructional  overviews  developed  are  available  on  videotape  and  on  an 
Internet  Web-site  to  inform  senior  leaders  and  unit  trainers  about  the  capabilities  of  the  CCTT 
and  structured  training.  This  report  documents  the  methods  and  lessons  learned  in  CITT  design, 
and  in  developing  and  formatively  evaluating  the  prototype.  It  will  be  useful  to  individuals  and 
agencies  involved  in  the  development  and  implementation  of  Army  training  management 
systems  for  live,  virtual,  or  constructive  training  environments. 


\£  (X-  9  ^  'i  ■  •>  yvc  a  ‘ 


f  A  M.  SIMUTIS 
Pechnical  Director 
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THE  COMMANDERS’  INTEGRATED  TRAINING  TOOL  FOR  THE  CLOSE  COMBAT 
TACTICAL  TRAINER:  DESIGN,  PROTOTYPE  DEVELOPMENT,  AND  LESSONS 
LEARNED 

EXECUTIVE  SUMMARY  _ 


Research  Requirement: 

The  U.S.  Army  is  currently  fielding  the  Close  Combat  Tactical  Trainer  (CCTT)  as  the 
first  member  of  the  Combined  Arms  Tactical  Trainer  (CATT)  family.  The  CCTT  provides  a 
virtual  environment  supporting  the  collective  training  of  armored  and  mechanized  infantry  units. 
To  maximize  its  effectiveness,  the  CCTT  will  be  fielded  as  a  complete,  integrated  training 
system,  i.e.,  in  addition  to  the  basic  hardware  and  software  that  comprise  the  system,  it  will  also 
provide  the  tools  required  to  enable  its  users  to  achieve  maximum  benefit  from  its  use.  As  CCTT 
training  tools,  techniques,  and  procedures  have  evolved,  the  need  has  increased  for  integrating 
them  so  that  commanders  and  other  unit  trainers  can  access  and  use  them  readily  and  effectively. 
Such  an  integrating  system  or  tool  should:  (a)  provide  trainers  with  ready  access  to  all  the 
information  and  methods  they  need  to  exploit  the  emerging  capabilities  of  CCTT;  (b)  be 
compatible  with  Army  training  management  information  systems  and  databases;  (c)  lead  users  to 
effective  and  efficient  methods  for  developing  and  implementing  training  by  providing  ready 
access  to  available  exercises,  associated  Training  Support  Packages  (TSPs)  and  other  materials; 
(d)  provide  users  with  an  understanding  of  and  means  to  apply  a  structured  approach  to  meeting 
training  requirements;  and  (e)  address  the  training  of  digital  forces. 

In  October,  1997  a  project  was  initiated  to  address  the  design  and  development  of  a  tool 
having  the  characteristics  described  above.  The  tool  is  the  Commanders’  Integrated  Training 
Tool  (C1TT)  for  the  CCTT.  The  overall  purpose  of  the  project  was  to  design  the  CITT  system  to 
provide  unit  commanders  and  other  unit  trainers  with  the  capability  to  maximize  the 
effectiveness  of  their  unit  training  in  the  CCTT  virtual  trainer.  The  CITT  design  would  allow 
commanders  to  select  existing  training  exercises  that  match  their  unit’s  needs,  and  if  no  such 
exercises  exist,  to  modify  existing  exercises  or  develop  new  ones.  Additional  purposes  of  the 
project  were  to  provide  an  Instructional  Overview  (10)  of  CCTT  including  detailed  coverage  of 
the  principles  of  structured  training  for  inclusion  in  the  CITT  and  to  serve  as  the  basis  for  the 
development  of  two  videotapes;  to  develop  a  prototype  of  the  CITT  as  a  proof  of  concept  and  to 
refine  the  prototype  through  formative  evaluation;  to  develop  implementation  methods  and 
fielding  recommendations  for  the  CITT;  and  to  record  and  document  lessons  learned  for 
application  in  similar  projects. 

Procedure: 

The  project  objectives  were  accomplished  through  the  completion  of  six  research  and 
development  tasks.  In  the  first  month  of  the  project  a  comprehensive  research  and  development 
plan  was  produced  and  submitted  for  ARI  approval.  Following  approval  work  began  on 
designing  the  Instructional  Overview  incorporating  CCTT  into  a  training  strategy  and  addressing 
CCTT  training  capabilities  along  with  methods  for  exploiting  them.  The  outcomes  of  this  task 
served  as  the  basis  for  the  Learn  About  CCTT  module  of  the  CITT  prototype  and  for  the 
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development  of  two  videos.  Simultaneously  with  10  design,  design  of  the  CITT  was  initiated 
and  proceeded  throughout  the  duration  of  the  project.  CITT  design  was  accomplished  using 
Army  approved  design  and  modeling  tools  and  procedures.  When  CITT  design  was  sufficiently 
complete,  work  on  the  development  of  a  prototype  CITT  was  initiated.  The  prototype  served  as 
a  “proof  of  concept”  and  was  produced  in  two  versions — a  standalone  version  and  a  distributed 
internet  accessible  version,  although  the  standalone  version  had  substantially  greater 
functionality.  Formative  evaluation  of  the  prototype  was  conducted  at  Fort  Hood  and  Fort  Knox, 
and  the  results  were  used  to  refine  the  CITT  prototype.  Refinement  of  the  prototype  included  the 
production  of  user  and  administrator  documentation  for  both  the  standalone  and  distributed 
versions.  In  addition,  a  proposed  implementation  strategy  and  fielding  plan  for  the  CITT  was 
produced.  The  final  activities  of  the  project  involved  documenting  lessons  learned  during 
development  of  the  CITT  along  with  recommendations  relating  to  implementation  and  fielding. 

Findings: 

Overall,  the  project  was  completed  successfully.  CITT  design,  the  primary  objective  of 
the  project,  was  completed  and  documented  from  the  standpoint  of  the  unit  trainer  as  the  CITT 
‘To-Be”  1.0.  A  prototype  CUT  in  two  versions  (standalone  and  distributed)  was  developed, 
with  the  standalone  version  including  greater  functionality.  The  prototype  included  those 
functions  of  CITT  design  which  were  feasible  during  the  course  of  the  project.  10  development 
included  the  production  of  two  videotapes  as  well  as  the  Learn  About  CCTT  function  of  the 
prototype.  Evaluation  activities  occurred  throughout  the  project  and  included  formative 
evaluation  of  the  prototype.  Refinement  of  the  prototype  was  based  on  findings  of  formative 
evaluation.  Approximately  forty  percent  of  the  findings  were  able  to  be  included  in  the  refined 
CITT;  the  remainder  are  included  in  the  report  as  future  desired  functionalities  of  CITT.  A 
fielding  plan  was  designed  and  developed  which  provides  several  alternative  fielding  strategies 
for  the  CITT,  and  lessons  learned  were  documented. 

Utilization  of  Findings: 

The  specific  audiences  who  will  find  the  information  contained  in  this  report  beneficial 
include:  (a)  designers  and  developers  who  continue  further  development  of  the  CITT,  (b) 
training  unit  and  CCTT  training  site  personnel,  (c)  simulation  system  developers,  and  (d)  any 
member  of  the  U.S.  Army  who  wants  to  better  understand  the  TSP  development  process.  The 
primary  product  of  the  CITT  Project,  the  CUT  design,  is  fully  documented  and  can  be  used  as 
the  basis  for  the  development  of  an  integrated  training  tool  under  any  of  several  fielding 
alternatives  developed  as  part  of  the  project.  In  addition,  the  videotapes  and  instructional 
overview  could  be  fielded  independently  to  provide  a  wide  variety  of  users  with  information  on 
the  CCTT  and  the  use  of  structured  training. 
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THE  COMMANDERS’  INTEGRATED  TRAINING  TOOL  FOR  THE 
CLOSE  COMBAT  TACTICAL  TRAINER: 

DESIGN,  PROTOTYPE  DEVELOPMENT,  AND  LESSONS  LEARNED 

INTRODUCTION 

The  U.S.  Army  is  currently  fielding  the  Close  Combat  Tactical  Trainer  (CCTT)1  as  the 
first  member  of  the  Combined  Arms  Tactical  Trainer  (CATT)  family.  The  CCTT  provides  a 
virtual  environment  supporting  the  collective  training  of  armored  and  mechanized  infantry  units 
including  combat  support  and  combat  service  support  elements.  To  maximize  its  effectiveness, 
the  CCTT  will  be  fielded  as  a  complete,  integrated  training  system,  i.e.,  in  addition  to  the  basic 
hardware  and  software  that  comprise  the  system,  it  will  also  provide  the  tools  required  to  enable 
its  users  to  achieve  maximum  benefit  from  its  use.  Several  such  tools  are  currently  available  in 
various  stages  of  development.  One  is  an  interactive  courseware  product  called  Education  of 
CCTT  through  Computer  Assisted  Training  Technology  (EDUCCATT)  being  developed  by  the 
Project  Manager  (PM)  for  CATT  to  train  unit  personnel  on  the  operation  of  CCTT  workstations 
supporting  the  execution  of  training  exercises.  Another  is  a  set  of  more  than  fifty  training 
exercises  and  training  support  packages  (TSPs)  that  were  developed  by  the  Army  Research 
Institute’s  Armored  Forces  Research  Unit  (ARI AFRU)  at  Fort  Knox,  KY,  in  conjunction  with 
PM  CATT  and  the  Training  and  Doctrine  Command  (TRADOC)  System  Manager  (TSM)  for 
CATT  under  the  Structured  Training  for  Units  in  the  Close  Combat  Tactical  Trainer 
(STRUCCTT)  (Flynn,  Campbell,  Myers,  &  Burnside,  1998)  and  the  follow-on  STRUCCTT-2 
projects  (Deatz,  Forrest,  Holden,  Sawyer,  Britt,  &  Gray,  1998). 

As  CCTT  training  tools,  techniques,  and  procedures  have  evolved,  the  need  has  increased 
for  integrating  them  so  that  commanders  and  other  unit  trainers  can  access  and  use  them  readily 
and  effectively.  Such  an  integrating  system  or  tool  should:  (a)  provide  trainers  with  ready  access 
to  all  the  information  and  methods  they  need  to  exploit  the  emerging  capabilities  of  CCTT;  (b) 
be  compatible  with  Army  training  management  information  systems  and  databases;  (c)  lead  users 
to  effective  and  efficient  methods  for  developing  and  implementing  training  by  providing  ready 
access  to  available  exercises,  associated  TSPs  and  other  materials;  (d)  provide  users  with  an 
understanding  of  and  means  to  apply  a  structured  approach  to  meeting  training  requirements;  and 
(e)  address  the  training  of  digital  forces. 

In  October,  1997,  ARI  AFRU  initiated  a  project  to  address  the  design  and  development  of 
a  tool  having  the  characteristics  described  above.  That  tool  is  the  Commanders’  Integrated 
Training  Tool  (CITT)  for  the  CCTT.  The  project  involved  the  design  of  the  CITT  as  its  primary 
focus  and  employed  standard  accepted  modeling  tools  and  methods  to  capture  the  design  from 
the  point  of  view  of  commanders  and  other  unit  trainers.  Secondary  goals  of  the  project  involved 
development  of  an  instructional  overview  (10)  of  the  CCTT  including  production  of  two  video 
tapes,  the  development  of  a  CITT  prototype  to  demonstrate  in  concept  the  feasibility  of  such  a 
tool  for  use  by  unit  commanders  and  other  unit  trainers,  and  formative  evaluation  of  the 
prototype.  Throughout  the  project,  the  development  team  collected  data  on  the  modeling  and 
development  processes  and  those  data  appear  in  this  report  as  the  basis  for  lessons  learned. 


1  A  list  of  all  acronyms  used  in  this  report  is  included  in  Appendix  A. 
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Purpose  of  the  Report 


The  purpose  of  this  report  is  to  describe  the  research  methods  and  outcomes  of  the  project 
particularly  as  related  to  the  primary  project  objective  of  designing  the  CITT.  The  report  will 
also  describe  the  activities  and  procedures  involved  in  completing  additional  project  objectives 
such  as  10  and  prototype  development.  The  report  provides  an  in-depth  understanding  and 
description  of  the  methodology  and  tools  used  to  produce  and  document  the  CITT  design 
including  a  discussion  of  the  differences  between  the  CITT  design  and  the  implementation  of  the 
design  in  the  CITT  prototype.  The  report  describes  how  the  project  team  was  organized  and 
integrated  to  achieve  the  project  goals,  the  methods  and  procedures  employed  to  complete 
project  activities,  and  the  evaluation  conducted  throughout  the  project,  including  quality 
assurance  activities  and  formative  evaluation  of  the  CITT  prototype.  Team  members  and  other 
individuals  associated  with  the  project  contributed  the  lessons  learned.  By  studying  this  report, 
the  reader  should  gain  a  thorough  understanding  of  the  goals  of  the  project,  the  project’s 
development  and  evolution,  how  the  team  dealt  with  challenges  faced  in  the  project,  and  how  the 
lessons  learned  can  be  applied  to  similar  projects. 

Organization  of  the  Report 
This  report  is  organized  as  follows: 

1 .  The  Project  Background  and  Need  section  describes  the  need  for  the  CITT  and  the  overall 
purpose  of  the  project  including  a  statement  of  the  project  objectives. 

2.  The  Research  Methodology  section  describes  the  activities  that  were  accomplished  to 
achieve  the  project  objectives  organized  by  major  project  tasks.  It  describes  the  results  of  the 
activities  including  a  description  of  the  CITT  design,  a  description  of  the  10  content  and 
development,  a  description  of  the  development  of  the  CITT  prototype  in  Standalone 
(CITTSA)  and  Distributed  (CITTDT)  versions,  and  a  description  of  the  implementation 
strategy  and  fielding  plan  including  recommendations  of  the  project  team  for  fielding  the 
CITT  as  ultimately  envisioned  or  designed.  This  includes  a  description  of  how  the  CITT 
could  be  integrated  into  other  existing  and  planned  Army  training  information  systems. 

3.  The  Project  Evaluation  section  includes  a  discussion  of  the  quality  assurance/control 
activities  conducted  throughout  the  project  and  concludes  with  a  description  of  the  formative 
evaluation  of  the  CITT  prototype  including  the  refinements  made  to  the  prototype  based  on 
the  results  of  formative  evaluation. 

4.  The  Lessons  Learned  section  describes  findings  from  the  project  that  are  relevant  to  future 
design  and  development  efforts. 

PROJECT  BACKGROUND  AND  NEED 

The  need  for  a  CITT  has  arisen  from  several  related  thrusts.  First  is  the  need  to  ensure 
that  the  training  that  units  receive  in  CCTT  is  matched  as  closely  as  possible  to  their  needs,  and 
that  it  has  maximum  effectiveness  and  efficiency.  Next  is  the  increasing  use  of  structured 
training  in  a  virtual  environment  as  a  vehicle  for  training  mission  essential  tasks.  Finally,  the 
need  exists  to  provide  easy  access  to  other  Army  training  management  and  database  systems 
such  as  the  Training  Module  (TRAMOD)  Executive  Management  Information  System 
(TEXMIS)  or  the  Army  Doctrine  and  Training  Digital  Library  (ADTDL)  and/or  integration  into 
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existing  training  systems  such  as  the  Standard  Army  Training  System  (SATS)  or  the  Automated 
Systems  Approach  to  Training  (ASAT). 

Training  in  CCTT 

The  CCTT  is  a  Distributed  Interactive  Simulation  (DIS)  compliant  system  designed  to 
facilitate  the  training  of  collective  armor  and  mechanized  infantry  tasks  at  the  platoon  through 
battalion  task  force  level.  CCTT  is  composed  of  armored  vehicle  manned  module  simulators  as 
well  as  Semi-Automated  Forces  (SAF),  Combat  Support  and  Combat  Service  Support  (CSS) 
workstations,  computer  networks  and  protocols,  and  After-Action  Review  (AAR)  systems. 
CCTT  provides  a  virtual  environment  for  training  that  can  be  used  in  a  variety  of  ways. 
According  to  the  Simulation,  Training  and  Instrumentation  Command  (STRICOM),  “CCTT  is 
the  first  of  the  Combined  Arms  Tactical  Trainer  (CATT)  family  of  virtual  trainers.  CCTT  will 
train  Armor,  Cavalry,  and  Mechanized  Infantry  platoons  through  Battalion/Task  Force  on  their 
doctrinal  Mission  Training  Plan  collective  tasks.”  (STRICOM,  1998a) 

Figure  1  illustrates  the  typical  CCTT  system  and  functions.  Manned  modules  consist  of 
high  fidelity,  full-crew  replications  oftheMlAl/MlA2  Abrams  Main  Battle  Tank,  the 
M2A2/M3A2  Bradley  Fighting  Vehicle,  the  Ml  13 A3  Armored  Personnel  Carrier  (APC),  the 


INFANTRY  ARMOR 


Figure  1.  The  Close  Combat  Tactical  Trainer. 

M981  Fire  Support  Team  Vehicle  (FIST-V)  and  the  High  Mobility  Multipurpose  Wheeled 
Vehicle  (HMMWV).  Additionally,  CCTT  includes  a  manned  simulator  that  allows  the 
command  element  of  dismounted  infantry  platoons  (the  Dismounted  Infantry  Module  or  DIM)  to 
participate  in  exercises  on  the  synthetic  battlefield.  Workstation  components  include  multiple 
operations  center  or  unit  support  workstations  including  the  Combat  Trains  Command  Post 
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(CTCP),  Field  Artillery  Battalion  Tactical  Operations  Center  (FABTOC),  Combat  Engineer 
Support  (CES),  Fire  Direction  Center  (FDC),  Unit  Maintenance  Collection  Point  (UMCP),  and 
Tactical  Air  Control  Party  (TACP).  Control  consoles  include  the  Master  Control  Console 
(MCC),  Maintenance  Console  (MC),  and  the  AAR  Console.  CCTT  includes  SAF  workstations 
which  control  the  Computer  Generated  Forces  (CGF)  subsystem  simulating  opposing  forces 
(OPFOR)  as  well  as  friendly  forces  (BLUFOR)  for  a  training  exercise.  Finally,  CCTT  includes 
the  unit  Tactical  Operations  Center  (TOC)  and  Higher  Headquarters  (HHQ).  CCTT  provides  a 
realistic  virtual  environment  in  which  units  train  on  and  perform  tasks  in  order  to  successfully 
accomplish  their  collective  missions. 

An  important  feature  of  the  CCTT  involves  training  customization.  That  is,  units  can,  at 
least  to  some  degree,  determine  the  training  experience  they  will  receive  in  CCTT.  At  present, 
however,  the  mechanism  for  customizing  involves  working  directly  with  CCTT  site 
administrators  who  have  the  knowledge  and  skills  necessary  to  modify  training  exercises.  A  tool 
to  provide  unit  commanders  with  the  information  necessary  to  plan  and  customize  their  training 
will  have  great  benefit. 


Structured  Training 

Structured  training  is  an  integral  component  of  the  Army’s  Systems  Approach  to  Training 
(SAT)  as  described  in  TRADOC  Regulation  350-70  (U.S.  Army  Training  and  Doctrine 
Command,  1995).  Structured  training  “provides  mission-based  task-focused  exercises  for  units 
or  staff  groups.  The  exercises  are  deliberately  designed  to  assure  that  specific  situations  and 
events  occur  providing  appropriate  conditions  for  practicing  performance  of  particular  tasks,  sub¬ 
tasks,  or  actions”  (Bessemer  &  Myers,  1998). 

As  summarized  by  Bessemer  and  Myers  (1998),  structured  training  exercises  include  a 
number  of  key  characteristics. 

1.  Scenario  Embedded.  Exercises  are  embedded  in  a  complete  mission  scenario  providing  a 
meaningful  context. 

2.  Execution  Focused.  Actions  required  to  execute  the  exercise  are  emphasized. 

3.  Mission  Segmented.  Platoon  or  company  exercises  are  of  limited  duration  allowing  for 
frequent  after  action  reviews  (AARs). 

4.  Task  Driven.  Exercise  events  elicit  performance  of  specific  tasks  that  support  the  training 
objectives  allowing  for  observation  and  evaluation  based  on  specified  task  standards. 

5.  Compressed  Time.  Maximum  training  is  delivered  in  the  available  training  time. 

6.  Fully  Supported.  Training  is  fully  supported  by  providing  unit  preparation  guidance,  detailed 
event  guides,  and  materials  for  observer-controllers  (O/Cs)  and  simulator  operators  who 
conduct  the  training.  The  vehicle  for  accomplishing  this  is  the  Training  Support  Package 
(TSP). 
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7.  Standardized  Library.  TSPs  are  available  to  support  execution  of  common  tasks  in  the 
Mission  Essential  Task  List  (METL)  for  particular  units. 

Exercises  are  designed  to  a  set  of  initial  conditions  (i.e.,  mission,  enemy,  troops,  terrain, 
and  time  available  [METT-T])  to  ensure  that  specific  conditions  and  events  occur  to  reinforce 
learning  and  build  on  prior  experience.  Structured  exercises  focus  on  a  small  number  of  critical 
Army  Training  and  Evaluation  Plan  -  Mission  Training  Plan  (ARTEP-MTP)  tasks  allowing  the 
unit  leader  to  observe  and  evaluate  performance  based  on  selected  task  standards.  All  exercises 
permit  standardized  (that  is  repeatable)  implementation  of  these  tasks,  task  steps,  and  conditions. 
Tasks  are  cued  consistently  which  allows  the  unit  to  train  and  retrain  to  achieve  task  proficiency 
at  low  cost  and  without  extensive  resources. 

In  addition  exercises  are  designed  with  increasing  levels  of  difficulty  to  promote  a 
crawl/walk/run  training  progression.  Unit  leaders  can  execute  an  exercise  under  a  simple  (crawl) 
condition  then  graduate  the  unit  to  more  difficult  conditions  when  the  unit  masters  the  exercise 
tasks  by  varying  the  conditions  under  which  the  task  is  performed  (e.g.,  enemy,  terrain,  and 
environmental  conditions.) 

Structured  training  is  implemented  through  the  use  of  training  support  packages  (TSPs) 
that  include  all  the  materials  necessary  to  organize  and  conduct  training  and  provide  focused 
feedback.  TSPs  provide  standard  unit  preparation  materials,  tactical  materials,  and  exercise 
control  materials  and  instructions  for  the  O/C  and  support  personnel.  The  O/C  and  support 
personnel  use  the  control  materials  to  direct  the  exercise,  coach  the  unit  during  the  exercise,  and 
provide  focused  feedback  following  the  exercise.  Training  is  turn-key.  Units  schedule  the 
training,  receive  and  use  the  pre-exercise  materials  to  rehearse,  arrive  at  the  training  site  and 
execute  the  mission(s),  and  receive  focused  feedback  on  their  performance. 

By  using  a  structured  approach  to  training,  unit  leaders  focus  on  tasks  from  the  unit’s 
METL,  identify  and  correct  training  deficiencies,  reinforce  performance  of  specific  tasks,  and 
assess  the  unit’s  readiness  level.  Structured  training  also  allows  unit  leaders  to  conduct  more 
frequent  after  action  reviews,  so  units  can  discuss  the  battle  while  the  events  and  actions  are  still 
fresh  in  their  minds. 

Structured  training  exercises  and  TSPs  have  been  developed  under  a  number  of  ARI 
directed  projects  including  the  Virtual  Training  Program  (VTP)  (Hoffman,  Graves,  Koger, 

Flynn,  &  Sever,  1995),  STRUCCTT  (Flynn,  et.  al.,  1998),  and  STRUCCTT-2  (Deatz,  et.  al., 
1998).  The  initial  STRUCCTT  projects  enhanced  the  capabilities  of  the  CCTT  simulation 
system  by  creating  structured  exercises  and  TSPs  to  exploit  the  CCTT  system  and  training 
capabilities  much  as  the  VTP  did  for  the  Simulation  Networking  (SIMNET)  system. 

To  date,  over  60  structured  TSPs  have  been  developed  for  CCTT  including  a  set  of  40 
initial  exercises  for  tank  and  mechanized  infantry  platoons  and  company  teams.  More  recently, 
exercises  have  been  developed  for  heavy  cavalry  units.  Although  this  may  appear  to  provide  a 
substantial  library,  when  one  considers  the  number  of  scenarios  required  to  train  at  each  echelon, 
across  each  type  of  unit,  for  each  type  of  mission,  at  differing  levels  of  intensity,  and  with 
varying  environmental  and  battlefield  conditions,  the  number  of  possible  exercises  is  practically 
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limitless.  Providing  the  commander  with  a  tool  to  customize  TSPs  is  another  important  function 
of  the  CITT. 

Wilkinson  (in  preparation)  has  recently  described  the  need  for  such  a  tool  that  will  allow 
unit  commanders  to  fully  exploit  CCTT  while,  at  the  same  time,  fulfilling  their  responsibilities  as 
unit  trainers.  ‘This  tool  is  envisioned  to  be  a  system  that  serves  as  a  repository  for  and  supports 
the  development  of  structured  training  scenarios.  Such  a  system  must  have  real-time  access  to 
information  on  the  appropriate  Mission  Training  Plans  (MTPs),  the  training  system,  task 
trainability  codes2  and  any  previously  developed  structured  scenarios.  This  system  must  support 
the  modification  of  existing  scenarios  as  well  as  the  development  of  new  scenarios.  This  system 
may  also  be  the  mechanism  for  actually  building  the  data  file  that  inputs  exercise  startup  data 
into  the  CCTT  system  or  any  other  training  system.” 

Wilkinson  recognizes  a  number  of  important  needs  that  such  a  system  should  address: 

1 .  It  should  assist  trainers  in  selecting  exercise  scenarios  based  on  the  unit’s  training  needs  if 
those  exercises  exist. 

2.  It  should  assist  trainers  in  modifying  existing  exercises  so  that  they  more  closely  match  the 
unit’s  training  needs. 

3.  It  should  assist  trainers  in  creating  new  exercises,  if  necessary,  that  take  advantage  of  and 
provide  all  of  the  benefits  of  structured  training.  This  should  be  accomplished  without  the 
necessity  of  the  unit  commander  being  a  subject  matter  expert  (SME)  in  developing 
structured  training  exercises. 

4.  It  should  train  the  trainers  in  how  to  exploit  the  capabilities  of  CCTT. 

5.  It  should  provide  each  trainer  the  same  level  of  information  on  system  capabilities  that  the 
CCTT  SMEs  used  to  develop  the  existing  library  of  exercises.  This  will  assist  innovative 
trainers  to  see  opportunities  to  use  CCTT  to  train  desired  tasks  effectively. 

In  short,  according  to  Wilkinson  (in  preparation),  “What  should  be  clear. .  .is  that  one 
cannot  afford  a  piecemeal  approach  to  TSPs  for  CCTT  or  any  training  system.  Units  require  a 
total  system  that  permits  trainers  to  fully  exploit  it. . .” 

On  the  other  hand,  full  exploitation  that  provides  for  customization  of  exercises  could 
lead  to  a  potential  conflict  between  structured  training  and  customized  training.  Modifications  to 
a  structured  training  exercise  could  easily  lead  to  non-structured  training.  This  is  particularly 
true  if  the  modifications  occur  “on  the  fly.”  To  safeguard  against  this,  it  is  extremely  important 
that  the  training  tool  described  by  Wilkinson  include  the  cognitive  support  (i.e.,  what  is 
structured  training,  how  is  it  applied,  why  is  it  important,  how  is  it  developed,  etc.)  It  is  also 
important  that  the  exercise  development  process  built  into  the  tool  lead  the  user,  to  the  maximum 
extent  possible,  to  develop  structured  training. 


2  Also  known  as  Task  Performance  Support  Codes  (TPSC).  TPSCs  describe  the  relative  capacity 
of  CCTT  to  support  collective  training  of  MTP  tasks  for  all  Battlefield  Operating  Systems 
(BOSs). 
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While  much  is  still  being  learned  regarding  structured  training  including  how  to  convey 
the  concepts  and  principles  to  a  wide  audience  of  potential  unit  trainers,  the  design  of  the  tool  in 
the  current  project  relied  on  the  combined  years  of  experience  of  members  of  the  development 
team  both  to  identify  and  define  the  cognitive  component  and  to  ensure  that  it  is  built  into  the 
TSP  development  process.  In  addition,  the  tool  will  not  be  developed  to  be  used  “on  the  fly.” 
Rather,  it  will  be  designed  to  be  used  as  part  of  a  carefully  planned,  deliberate  training  process 
which  provides  flexibility  to  the  unit  trainer  while  preserving  structured  training  principles. 

Army  Training  Systems 

Currently,  the  U.S.  Army  Training  Support  Center  (ATSC)  is  the  executive  agent  for  the 
Army  Training  Information  Management  Program  (ATIMP).  Program  activities  of  ATSC 
support  the  vision,  goals,  and  strategic  direction  of  the  Department  of  Army  (DA)  in  accordance 
with  the  Army  Information  Resources  Management  Program  (AR  25-1,  1997a).  The  program 
mission  is  to  provide  Army-wide  focus,  guidance,  and  oversight  to  the  development,  integration, 
and  operation  of  training  information  systems.  The  ATIMP  provides  a  management  and  support 
infrastructure  to  enhance  the  coordination  of  system,  process,  and  data  integration  and  to 
preclude  the  development  of  unnecessary  or  redundant  training  business  processes,  business 
rules,  and  information  systems. 

The  ATIMP  systems  provide  a  starting  point  for  commanders  and  unit  trainers  to  obtain 
information  and  plan  training  events.  What  is  needed  is  a  system  capable  of  “bringing  the 
systems  together”  to  provide  an  effective  and  efficient  method  for  developing  training  materials 
specifically  for  the  CCTT.  This  is  another  important  function  of  the  CITT. 

PROJECT  PURPOSE 

The  overall  purpose  of  the  CITT  Project  was  to  design  the  CITT  system  to  provide  unit 
commanders  and  other  unit  trainers  with  the  capability  to  maximize  the  effectiveness  of  their  unit 
training  in  the  CCTT  virtual  trainer.  The  CITT  design  would  allow  commanders  to  select 
existing  training  exercises  that  match  their  unit’s  needs,  and  if  no  such  exercises  exist,  to  modify 
existing  exercises  or  develop  new  ones.  Additional  purposes  of  the  project  were  to  provide  an 
IO  of  CCTT  including  detailed  coverage  of  the  principles  of  structured  training  for  inclusion  in 
the  CITT  and  to  serve  as  the  basis  for  the  development  of  two  videotapes;  to  develop  a  prototype 
of  the  CITT  as  a  proof  of  concept  and  to  refine  the  prototype  through  formative  evaluation;  to 
develop  implementation  methods  and  fielding  recommendations  for  the  CITT;  and  to  record  and 
document  lessons  learned  for  application  in  similar  projects. 

These  purposes  were  accomplished  through  the  completion  of  six  research  and 
development  tasks: 

Task  1:  Prepare  a  comprehensive  research  and  development  plan. 

Task  2:  Design  an  IO  addressing  incorporation  of  CCTT  into  a  training  strategy  and  CCTT 
training  capabilities  along  with  methods  for  exploiting  them. 

Task  3:  Design  a  complete  CITT  incorporating  the  CCTT  instructional  overview. 
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Task  4:  Develop  a  prototype  CITT  and  refine  it  through  formative  evaluation. 

Task  5:  Develop  an  implementation  strategy  and  fielding  plans/methods  for  the  CITT. 

Task  6:  Document  lessons  learned  during  development  of  the  CITT  along  with 

recommendations  relating  to  CITT  implementation  and  fielding. 

RESEARCH  METHODOLOGY 
Overview 

The  objectives  and  scope  of  the  CITT  Project  required  the  members  of  the  project  to 
work  efficiently.  Because  production  of  the  CITT  products  required  personnel  with  diverse 
knowledge  and  skills,  and  because  the  project  required  frequent  and  open  communication  among 
team  members,  a  flat  organization  was  created  to  accomplish  the  tasks  and  produce  the 
deliverables  of  the  CITT  research  and  development  effort.  Under  this  organization,  all  team 
members  reported  directly  to  the  Project  Leader,  and,  although  the  team  was  organized  loosely 
around  the  major  project  activities,  the  entire  team  participated  in  most  planning  and  decision¬ 
making  activities.  The  structure  of  the  development  team  emerged  because  of  the  need  to 
integrate  the  efforts  of  CCTT  and  structured  training  experts  along  with  training  development, 
database,  information  management,  evaluation,  and  Internet  experts  through  all  phases  of  the 
project.  The  flat  organization  also  provided  for  the  effective  coordination  of  the  efforts  of 
Human  Resources  Research  Organization  (HumRRO),  the  prime  contractor,  and  its 
subcontractors,  Raytheon  Inc.,  TRW,  VIA  Internet  Studios,  and  Litton-PRC.  Collectively  these 
five  organizations  formed  the  CITT  Project  team. 

Beginning  in  October,  1997,  the  initial  project  team  assembled,  consisting  of  the  project 
leader,  an  administrative  assistant,  a  multimedia  specialist,  a  systems  specialist,  an  Army  training 
specialist,  a  training  developer,  an  instructional  technologist,  a  web  site  development  specialist, 
and  a  training  systems  specialist.  In  mid-November,  the  team  added  a  program  evaluation 
specialist,  and  the  full  team  was  completed  in  January  with  the  addition  of  a 
database/management  information  systems  (MIS)  specialist.  The  team  remained  relatively  intact 
throughout  the  life  of  the  project.  The  stability  of  the  team  contributed  to  the  accomplishments 
of  the  project. 

Early  in  the  project,  the  team  made  critical  decisions  regarding  the  daily  operation  and 
interaction  of  the  team  based  on  the  need  to  integrate  the  members  and  their  diverse  backgrounds 
and  areas  of  expertise.  Developers  recognized  early  that  the  project  would  require  a  highly 
integrated  effort  to  ensure  the  timely  application  of  team  members’  expertise.  The  team  was 
encouraged  from  the  beginning  to  share  information,  ideas,  and  decisions  on  a  daily  basis.  Team 
members  became  resources  for  each  other. 

Initially,  the  team  collectively  identified  a  set  of  requirements  for  an  integrated  training 
tool  and  used  these  to  focus  the  research  on  existing  and  emerging  training  information 
management  systems.  Simultaneously,  the  team  worked  together  to  complete  the  CITT  Research 
Program  Plan  (1997),  and  when  the  plan  was  approved  by  the  Contracting  Officer’s 
Representative  (COR),  individuals  were  assigned  to  work  on  specific  tasks.  The  project  focused 
on  Tasks  2  and  3  initially  with  work  on  Task  4  to  begin  when  Task  3  was  approximately  80 
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percent  complete.  As  will  be  described  below,  this  requirement  was  modified  as  the  project 
proceeded  and  work  on  Task  4  began  well  before  Task  3  was  80  percent  complete.  Work  on 
Task  5  occurred  throughout  the  project  although  it  was  not  a  major  focus  until  late  in  the  project. 
Also  occurring  throughout  the  project  were  the  quality  assurance  activities  designed  to  monitor 
the  internal  processes  used  by  the  team  to  accomplish  its  tasks.  The  results  of  this  process,  as 
well  as  the  collective  input  of  the  project  team,  served  as  the  basis  of  the  lessons  learned  section 
of  this  report.  Throughout  the  project,  close  coordination  occurred  between  the  project  team  and 
the  COR  as  well  as  the  PM  and  TSM  CATT  representatives. 

The  remainder  of  this  section  provides  a  detailed  description  of  the  research  methodology 
and  activities  occurring  to  accomplish  each  project  task. 

Task  1:  Prepare  a  Comprehensive  Research  and  Development  Plan 

Beginning  in  early  October,  1997,  the  project  team  developed  a  comprehensive  research 
and  development  plan  based  on  the  CITT  Project  Statement  of  Work  (SOW)  (Department  of  the 
Army,  1997c)  and  the  Technical  Response  to  the  SOW.  After  thoroughly  analyzing  these 
products,  the  team  members  or  groups  of  team  members  were  assigned  to  research  the  detailed 
objectives  and  the  corresponding  project  tasks  to  determine  the  activities  required  to  complete 
them.  As  the  team  wrote  drafts  of  the  various  components  of  the  research  plan,  these  were 
circulated  among  the  team  members  for  review  and  response.  This  process  enabled  rapid 
completion  of  a  detailed  Research  Program  Plan  which  was  presented  to  the  COR  on 
November  25,  1997. 

One  major  change  from  the  original  SOW  and  the  Technical  Response  to  the  SOW 
should  be  noted  in  this  report.  The  proposed  project  schedule  added  slighdy  more  than  two 
months  to  the  period  of  performance  and  resulted  in  the  project  ending  in  early  December  1998, 
instead  of  October  1998.  This  extension  was  needed  to  allow  the  team  to  thoroughly  research 
the  capabilities  of  a  number  of  existing  and  emerging  training  information  management  systems 
being  developed  by  the  Army.  The  team  reasoned  that  these  systems  might  include  certain 
functions  required  within  the  CITT,  especially  those  related  to  TSP  development;  and  the  project 
team  recognized  the  importance  of  fully  understanding  the  capabilities  of  these  systems  so  the 
CITT  could  integrate  with  them  where  appropriate.  Additionally,  the  team  thought  it  necessary 
to  spend  more  time  than  originally  allowed  developing  the  CITT  prototype  since  many  of  the 
lessons  learned  during  development  would  apply  to  a  comprehensive  CITT  design.  The  COR 
approved  this  change  at  no  additional  cost  to  the  government. 


Task  2:  Design  an  Instructional  Overview  Addressing  Incorporation  of  CCTT  into  a 
Training  Strategy  and  CCTT  Training  Capabilities  Along  with  Methods  for  Exploiting  Them 

Following  approval  of  the  Research  Program  Plan,  a  subgroup  of  the  project  team  began 
work  on  Task  2.  The  purpose  of  the  10  was  to  provide  commanders  and  unit  trainers  of 
conventional  and  digital  units  (platoon  through  brigade  level)  with  the  tools,  techniques,  and 
procedures  needed  to  effectively  and  efficiently  plan  and  execute  CCTT  training. 
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A  major  project  constraint  concerned  the  ways  in  which  the  10  content  was  to  be 
provided  to  the  CUT  audiences.  The  SOW  specified  videotape  as  one  medium  for  presenting 
content,  but  at  two  distinct  levels:  brigade  commander  and  above,  and  brigade  commander  and 
below.  At  the  same  time,  the  project  required  the  integration  of  the  IO  content  information  into 
the  CUT  prototype.  As  illustrated  in  Figure  2,  a  comprehensive  analysis  of  relevant  information 
was  completed  to  identify  the  10  content.  Information  examined  included  the  TSM  CATT,  PM 
CATT,  and  STRICOM  Internet  sites,  the  CCTT  Interoperability  documentation  (STRICOM, 
1998b),  the  CCTT  Workstation  Operators  Guide  (WOG),  reports  from  the  STRUCCTT  (Flynn, 
et.  al.,  1998)  and  STRUCCTT-2  (Deatz,  et.  al.,  1998)  projects,  Field  Manuals  FM  25-100 
(Department  of  the  Army,  1988)  and  FM  25-101  (Department  of  the  Army,  1990),  and 
interviews  with  subject  matter  experts  on  CCTT  and  structured  training.  The  results  of  this 
analysis  served  as  the  basis  for  the  development  of  two  video  scripts:  ‘The  Senior  Leader’s 
Guide  to  CCTT  System  and  Training  Capabilities’*3  video  for  brigade  commanders  and  above, 
and  “The  Unit  Leader’s  Guide  to  Training  in  the  CCTT”4  video  which  provides  commanders  and 
unit  trainers  (platoon  through  brigade)  with  an  explanation  of  all  phases  of  CCTT  training 
including  how  to  plan,  prepare,  execute,  provide  feedback  during  AARs,  complete  post-training 
reports,  and  provide  follow  up  training  opportunities  available  in  the  CCTT.  Additionally,  it 
provides  a  brief  introduction  to  the  CITT.  The  team  designed  both  videos  to  complement 
existing  CCTT  promotional  videos  produced  by  PM  CATT. 

The  IO  content  also  served  as  the  basis  for  the  development  of  the  “Learn  About  CCTT” 
module  of  the  CUT  prototype.  The  "Learn  About  CCTT"  is  a  computer-based  information 
module  in  both  the  CUTSA  and  CITTDT  versions.  The  instructional  overview  and  Learn  About 
the  CCTT  module  of  the  CUT  are  based  on  structured  training  principles  and  the  overview  and 
train-the-trainer  materials  developed  as  part  of  STRUCCTT. 

The  remainder  of  the  Task  2  coverage  describes  the  methodology  for  designing  and 
developing  the  instructional  overview  videos  and  prototype  content. 


3  The  Senior  Leader’s  Guide  to  CCTT  System  and  Training  Capabilities  video  can  be  obtained 
from  the  Training  Support  Center  at  your  installation  by  ordering  TVT  17-221,  PIN  711132. 

4  The  Unit  Leader’s  Guide  to  Training  in  the  CCTT  video  can  be  obtained  from  the  Training 
Support  Center  at  your  installation  by  ordering  TVT  17-220,  PIN  711 131. 
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Figure  2.  The  instructional  overview. 

Design  a  Comprehensive  Instructional  Overview 

One  of  the  first  tasks  completed  in  the  design  of  the  Instructional  Overview  was  to 
investigate  the  CCTT  system  and  training  capabilities  including  those  available  at  the  Fort  Hood, 
TX,  site  through  1998.  The  investigation  included  a  review  of  CCTT  Interoperability 
Description  Document  and  training  information,  STRUCCTT  overview  and  unit  guidance 
materials,  as  well  as  Workstation  Operations  Guide  (WOG)  developed  by  the  PM  CATT.  It  also 
included  information  gathered  from  the  Internet  starting  with  the  STRICOM,  CCTT,  and  PM 
CATT  Internet  information  sites.  This  information  was  examined  and  reviewed  to  determine  the 
overall  system  and  training  capabilities  of  the  CCTT,  the  echelons  and  exercises  supported  by 
the  CCTT,  how  the  CCTT  supports  an  annual  training  strategy,  and  the  possible  uses  of  the 
CCTT  to  support  various  training  events  and  requirements. 

The  project  team  examined  the  experience  and  lessons  learned  from  the  STRUCCTT 
Projects  to  exploit  CCTT  training  capabilities,  especially  the  tools,  techniques,  and  procedures 
for  the  successful  planning,  preparation,  execution,  and  assessment  of  simulation  training  using  a 
structured  training  approach.  In  addition,  they  incorporated  information  gained  from  the  CCTT- 
Digital  (Dierksmeier,  Winsch,  Liebrecht,  Sawyer,  Quinkert,  &  Wilkinson,  1999)  research  project 
and  the  CCTT  XXI  vision  (LTC  Jeff  Wilkinson,  personal  communication,  May  22, 1998) 
developed  by  TSM  CATT. 

Project  team  members  used  their  experience  and  expertise  to  analyze  and  synthesize 
collected  information  to  produce  a  detailed  listing  of  the  content  of  the  instructional  overview. 
After  reviewing  and  researching  the  CCTT  system  and  training  capabilities,  the  next  step  was  to 
create  detailed  outlines  highlighting  the  major  points  of  information  to  be  presented  and  the 
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media  needed  to  illustrate  that  information  in  the  instructional  overview.  The  Team  identified 
the  major  high-level  topics,  broke  the  topics  into  groups,  then  subgroups  of  needed  information. 
The  next  step  was  to  visualize  the  text,  graphics,  audio,  and  video  that  would  best  enhance  the 
meaning  or  the  message  of  each  topic.  The  team  used  different  themes,  i.e.,  the  Army’s  Nine 
Principles  of  Training  and  “Captain  Smith  visiting  the  CCTT  Site”  to  prepare  for  his  scheduled 
CCTT  training  rotation,  to  reflect  the  interest  of  the  audience,  promote  the  understanding  of 
CCTT  tools,  techniques,  and  procedures,  and  support  Army  training  methodologies.  At  this 
point,  the  team  separated  video  design  and  computer-based  design,  at  least  conceptually,  and 
continued  to  work  on  each. 

Video-based  Design  and  Development. 

As  stated  previously,  two  instructional  videos  were  developed:  The  “Senior  Leader’s 
Guide  to  CCTT  System  and  Training  Capabilities”  geared  towards  brigade  commanders  and 
above,  and  the  “Unit  Leader’s  Guide  to  Training  in  the  CCTT”  for  unit  training  leaders  (platoon 
through  brigade). 

The  “Senior  Leader’s  Guide  to  CCTT  System  and  Training  Capabilities”  includes  the 
following  information: 

1.  How  the  CCTT  system  creates  a  realistic  and  challenging  training  environment. 

2.  How  the  CCTT  builds  on  the  experience  gained  from  the  SIMNET  system. 

3.  The  terrain  databases  available  in  the  CCTT  system. 

4.  The  types  of  environmental  conditions  simulated  in  the  CCTT  system. 

5.  The  major  components  of  the  CCTT  system. 

6.  The  types  of  manned  modules  and  workstations  available  for  training. 

7.  The  training  capabilities  of  the  CCTT  system. 

8.  The  locations  of  the  CCTT  Fixed  and  Mobile  Sites. 

9.  How  the  CCTT  can  support  an  annual  training  strategy  to  include  use  of  CCTT  before  or 
after  various  training  events  and  requirements. 

10.  How  the  CCTT  can  support  training  of  digital  units. 

11.  The  principles,  characteristics,  and  benefits  of  using  a  structured  approach  to  training  in  the 
CCTT. 

12.  The  components  and  materials  contained  in  a  CCTT  training  support  package. 

13.  The  types  of  structured  exercises  available  for  the  CCTT  system. 

14.  The  Commanders'  Integrated  Training  Tool  and  methods  for  accessing  it. 

The  “Unit  Leader’s  Guide  to  Training  in  the  CCTT”  includes  the  following  information: 

1 .  How  the  CCTT  system  creates  a  realistic  and  challenging  training  environment. 

2.  The  terrain  databases  available  in  the  CCTT  system. 

3.  The  types  of  environmental  conditions  simulated  in  the  CCTT  system. 

4.  The  major  components  of  the  CCTT  system. 

5.  The  types  of  manned  modules  and  workstations  available  for  training  including  combat, 
combat  support,  and  combat  service  support  elements. 

6.  The  principles,  characteristics,  and  benefits  of  using  a  structured  approach  to  training  in  the 
CCTT. 
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7.  The  components  and  materials  contained  in  a  CCTT  training  support  package. 

8.  The  types  of  tasks  trained  in  the  CCTT. 

9.  The  Commanders’  Integrated  Training  Tool  and  methods  for  accessing  it. 

10.  The  activities  involved  in  planning  for  CCTT  training  including  information  on  how  to  plan 
for  CCTT  training,  use  the  CITT  to  select  the  training,  review  the  support  requirements, 
coordinate  with  the  CCTT  Site,  and  the  types  of  supplies  and  equipment  needed. 

1 1 .  The  activities  involved  in  preparing  for  CCTT  training  including  conducting  troop  leading 
procedures,  familiarization  training,  and  attending  the  Site’s  Initial  Briefing. 

12.  The  activities  involved  in  executing  a  CCTT  exercise  including  the  tools,  techniques,  and 
procedures  for  executing  a  structured  training  exercise  in  the  CCTT.  This  also  highlights 
the  roles  and  responsibilities  of  the  training  unit,  the  observer/controller  (O/C),  and 
workstation  operators. 

13.  The  activities  involved  in  assessing  a  CCTT  exercise  including  the  tools,  techniques  and 
procedures  for  assessing  the  unit’s  task  performance. 

Upon  completion  of  the  outlines,  the  team  developed  scripts  to  provide  the  dialogue 
(narration)  and  visuals  as  well  as  instructions  for  producing  the  instructional  overview  (e.g., 
sound  effects,  timing,  camera  angles,  and  lighting).  In  addition  to  the  scripts,  they  created 
storyboards  for  those  scenes  having  multiple  events  to  illustrate  how  those  events  would  be 
included  in  the  video. 

Pre-production.  Once  the  design  was  approved  and  recommendations  incorporated,  the 
project  team,  with  the  support  of  the  Fort  Knox  Television  Division,  produced  the  instructional 
overview  videos.  One  of  the  advantages  of  partnering  with  Television  Division  was  that  both 
videos  would  be  readily  available  through  Army  distribution  channels.  The  project  team 
involved  Television  Division  in  the  earliest  stages  of  design  and  development  to  ensure  the 
overview  was  structurally  sound  and  presented  in  logical  sequence.  The  project  team  worked 
closely  with  Television  Division  during  the  pre-production,  production,  and  post-production 
phases  of  the  instructional  overview. 

During  the  pre-production  phase,  team  members  identified  the  resources  and  media 
requirements  needed  to  develop  the  instructional  overview.  These  requirements  specified  the 
most  appropriate  media  for  development  of  the  instructional  overview  including: 

1.  Video  footage. 

2.  Virtual  footage. 

3.  Text  On  Screen. 

4.  Computer  Animation  Graphics. 

5.  Tactical  Message  Traffic. 

Preparations  began  for  video  production  including  scheduling  the  CCTT  sites  at  Fort 
Knox  and  Fort  Hood,  coordination  with  the  STRUCCTT  Team  to  have  CCTT  electronic  files 
built  to  capture  virtual  footage,  and  discussions  with  the  training  unit  and  O/C  scheduled  for 
training  at  the  Fort  Knox  CCTT  site  to  ensure  the  intent  and  context  of  the  video  was  understood. 
Computer  animation  and  graphics  needed  during  production  were  also  designed. 
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Production.  Production  of  the  instructional  overview  videos  took  place  at  the  Fort  Knox 
and  Fort  Hood  CCTT  sites.  The  project  team  used  the  approach  to  creating  resource  reels 
illustrated  in  Figure  3.  Simultaneously  narration  was  produced,  graphics  were  developed  or 
selected,  and  video  clips  were  developed  or  selected.  These  were  used  to  produce  the  resource 
reel  from  which  the  CITT  videos  were  assembled. 

Fort  Knox  Television  Division  assisted  the  team  to  capture  the  necessary  video  and 
virtual  footage.  Although  a  majority  of  the  required  video  shots  were  staged,  the  team  decided  to 
capture  the  interactions  of  the  training  unit  and  the  O/C  with  minimal  interruptions  to  the  training 
process.  This  complex  process  included  two  cameras  set  up  to  focus  on  the  O/C  at  the  After 
Action  Review  Workstation,  two  cameras  located  inside  the  manned  module,  and  a  scan 
converter  employed  to  obtain  virtual  footage.  All  cameras  were  synchronized  to  capture 
interactions  simultaneously.  While  the  team  obtained  the  required  video  footage,  graphic  artists 
developed  the  computer  animation  and  graphics  needed  to  support  the  instructional  overview. 

After  obtaining  the  video  and  virtual  footage,  the  team  reviewed  the  material  to  select  the 
best  visuals  to  support  the  concept  of  the  video  and  the  narration.  Next,  they  created  the  resource 
reel.  For  each  scene,  the  team  selected  video  and/or  virtual  footage  and  placed  it  on  the  resource 
reel  for  assembly  with  the  narration.  Next,  they  merged  resource  reel  with  the  narration  and 
computer  animation  and  graphics,  and  assembled  them  onto  tape. 

Post-production.  During  post-production,  the  team  mixed  sound  bits  recorded  at  Fort 
Knox  and  Fort  Hood  with  background  music  and  sound  effects  to  achieve  the  best  possible 
clarity  and  quality.  Building  up  sound  effects  one  operation  at  a  time,  it  took  many  steps  to 
achieve  the  final  result.  Once  the  background  music  and  sound  effects  were  added,  they  were 
balanced  and  combined  on  a  single  audio  track.  When  this  was  completed,  work  began  on  the 
transitions  and  dissolves  between  each  scene.  The  final  step  involved  creating  the  master  tape 
then  “dubbing”  (copying)  the  master  for  distribution. 


Figure  3.  The  video  creation  process. 
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Computer-based  Design  and  Development 


In  addition  to  the  video-based  instructional  overview,  an  instructional  overview  for 
inclusion  in  the  CUTS  A  and  CITTDT  was  designed.  The  instructional  overview  serves  two 
purposes:  it  provides  unit  trainers  with  more  detailed  information  than  contained  in  the 
instructional  overview  videos  regarding  the  system  and  training  capabilities  of  the  CCTT  and 
methods  for  exploiting  them;  and  it  provides  unit  trainers,  site  staff,  workstation  operators,  and 
O/Cs  with  information  concerning  their  roles  and  responsibilities  in  planning,  preparing, 
executing  and  assessing  a  structured  training  exercise  in  the  CCTT. 

The  team  analyzed  the  instructional  overview  outlines,  scripts,  and  videos  to  determine 
the  information  to  include  in  the  computer-based  instructional  overview.  With  the  CUT 
prototype  design  (see  Task  4)  proceeding,  the  team  determined  that  this  information  would 
comprise  the  “Learn  About  the  CCTT”  module  of  the  CUT  prototype  for  both  the  CUTSA  and 
CITTDT. 

The  “Learn  About  the  CCTT’  module  for  CUT  system  contains  the  following 
information  topics: 

1.  Explore  the  CCTT:  An  overview  of  the  CCTT  system  describing  the  ‘Train  as  You  Fight” 
philosophy  including  a  general  reference  to  the  units  and  echelons  trained,  major  components 
of  the  CCTT,  features  of  the  CCTT,  and  the  standard  and  mobile  site  configurations. 

2.  Examine  the  CCTT  System  Capabilities:  An  overview  describing  the  system  capabilities  and 
role  of  the  major  components  of  the  CCTT  system.  It  also  provides  information  on  how  the 
major  components  interact  to  create  a  realistic  virtual  battlefield. 

3.  Examine  the  CCTT  Training  Capabilities:  An  overview  describing  the  training  capabilities 
of  the  CCTT  system  including  the  types  of  units  and  echelons  that  can  be  supported  in  the 
CCTT,  additional  training  possibilities,  and  how  the  CCTT  supports  an  annual  training 
strategy. 

4.  Learn  About  Structured  Training:  An  overview  describing  the  principles,  characteristics,  and 
benefits  of  using  a  structured  approach  to  training  in  the  CCTT.  It  also  provides  the 
components  and  materials  contained  in  a  CCTT  training  support  package. 

5.  Learn  About  the  Training  Process:  An  overview  describing  the  tools,  techniques,  and 
procedures  for  planning,  preparing,  executing,  and  assessing  a  structured  exercise  in  the 
CCTT. 

6.  Learn  About  CCTT  Exercises:  An  overview  describing  the  types  of  training  exercises 
supported  by  the  CCTT  (e.g..  Orientation  Exercises,  Situational  Training  Exercises,  Platoon 
Gunnery  Exercises,  Command  Field  Exercises,  and  Tactical  Exercises  Without  Troops). 

In  developing  the  computer-based  instructional  overview,  the  project  team  considered  how  the 
information  should  be  structured  for  maximum  effectiveness  when  presented  via  computer,  the 
content  layout,  and  how  users  might  choose  to  navigate  through  the  information. 

Structure.  The  computer-based  instructional  overview  is  designed  to  provide  users  with 
graduated  levels  of  information.  High-level  information  topics  (level  1)  are  identified,  broken 
into  groups  (level  2),  and  then  into  subgroups  of  information  (level  3).  Each  information  topic  is 
systematically  structured  and  labeled  so  that  the  user  can  selectively  view  or  skip  individual 
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topics  and/or  advance  to  a  topic  for  which  they  have  an  immediate  need  (see  Figure  4).  Each 
topic  is  composed  of  one  or  more  pages  of  information.  In  addition,  each  topic  has  hyperlinks  to 
optional  pages  containing  more  detailed  information  or  further  explanations  related  to  that  topic. 

Layout.  The  layout  of  the  instructional  overview  represents  a  balance  between  form 
(visual  appearance)  and  function  (easy  access  to  information).  The  team  intended  to  provide  a 
basic  layout  so  that  each  page  had  a  predictable  look  and  feel.  So  as  not  to  distract  the  user,  the 
background  color  remained  neutral.  Headers  with  prominent  titles  appeared  at  the  top  of  each 
page  to  assist  the  user  in  identifying  the  topic  and  level  of  information. 

Graphics  were  used  sparingly  and  only  when  necessary  to  assist  the  user  in  understanding 
the  content.  Graphics  were  in  the  form  of  two-dimensional  drawings  and  digitized  Graphic 
Interchange  Format  (GIF)  images.  In  addition,  the  team  included  demonstrations  of 
performances  as  embedded  videos  in  the  Standalone  version  to  provide  users  with  multimedia 
examples  of  structured  exercises  available  in  the  CCTT.5 
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♦  Icam  About  Structured  Trairong 

*  Leam  Abort  the  Training  Process 

♦  Leant  About  CCTT  Exercises 


Figure  4.  CITT  instructional  overview  structure. 

Navigation.  Users  were  given  several  options  for  navigating  through  the  instructional 
overview.  First,  by  using  standard  buttons  on  each  page  the  user  can  move  to  the  “Next”  page  or 
return  to  the  “Previous”  page  in  a  linear  sequence  (see  Figure  5).  Second,  the  prototype  design 


5  Demonstrations  of  performance  are  examples  in  the  form  of  digital  videos  which  show  users 
how  tasks  and  exercises  are  performed  in  the  CCTT. 
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allows  for  a  standard  browser,  and  all  browsers  provide  “Back”  and  “Forward”  buttons  that 
allow  the  user  to  move  backward  and  forward  through  pages  they  have  already  viewed.  Third, 
the  user  can  navigate  by  clicking  on  the  tree  diagram  shown  in  the  left-hand  frame  of  Figure  4 
with  each  topic  serving  as  a  link  enabling  users  to  move  to  any  topic,  or  page  within  a  topic,  in  a 
non-linear  sequence.  Fourth,  the  topics  include  numerous  embedded  hyperlinks  (see  Figure  5) 
which  provide  the  user  with  a  “threaded”  navigation  scheme;  i.e.,  the  user  can  examine  content 
related  to  a  topic  by  selecting  hyperlinks  to  related  information.  Finally,  the  user  has  available 
an  index  and  a  search  feature  allowing  the  user  to  select  or  type  in  a  keyword  or  phase  in  order  to 
view  the  desired  information. 

To  summarize,  based  on  the  extensive  body  of  information  on  CCTT,  structured  training, 
etc.,  assembled  for  Task  2,  the  team  members  designed  the  “Learn  About  CCTT”  component  of 
the  CITT  to  provide  the  user  with  all  of  the  information  he  or  she  will  require  to  make  effective 
use  of  training  in  the  CCTT.  The  team  also  designed  the  components  to  be  user  friendly  and  to 
allow  for  a  variety  of  user  styles.  Users  can  choose  among  several  different  methods  of 
navigating  through  the  content,  and  a  variety  of  instructional  modes  including  text,  graphics,  and 
video  are  used  to  support  learning. 
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Components  of  a  Training  Support  Package  j 


A  training  support  package  (TSP)  is  a  complete  package  of  integrated  training  materials  that  includes 
information  necessary  to  efficiently  and  effectively  train  ARTEP-MTP  tasks. 


A  CCTT  TSP  contains  the  following  components  and  materials: 


_  TSP  .. 
Components 

.  Materials 

1  Selection 

These  materials  assist  the  unit  leader  in  ; 
selecting  exercises  needed  to  support 
the  unit’s  training  plan  and  objectives. 

Exercise  Thumbnails  ! 

Materials 

Exercise  Outlines  : 

.-'...  V  i 

Tactical  Materials 

These  materials  provide  the  operational 
context  for  the  exercise. 

OPORDs 

Overlays 

Exercise  Materials 

These  materials  provide  the  information 
and  instructions  that  the  unit  will  need 
to  conduct  a  structured  exercise  in  the  :j 
CCTT,  It  also  contains  the  information  j 
and  ihsthictions  used  to  conduct  the  :-.  J 

aail  vj 

Exercise  TSP  that  includes: 

Event  Guides 

.  ‘.l\ 

Workstation  Execution  Guidelines 

AAR  Worksheets  Task  Charts 

■  .-•■  V  ■■■;•  >  ..  :  j 

Simulation 

These  materials  provide  the  information 
the  CCTT  Site  Staff  will  need  to  build 
and  proof  the  exercise  file. 

Plan  Sheets 

Materials 

Commo  Data  f 

Exercise  Files 

.  & 


Figure  5.  An  example  of  the  use  of  hyperlinks  for  navigation  within  the  instructional  overview. 
Hyperlinks  are  underlined  words  and  phrases. 
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Task  3:  Design  a  Complete  CITT  Incorporating  the  CCTT  Instructional  Overview 

This  section  provides  information  on  the  design  of  the  CITT.  It  includes  discussion  of 
the  needs  assessment  activities  related  to  Task  3,  the  definition  of  the  CITT  (hardware  and 
software)  requirements,  the  design  and  modeling  process,  and  the  actual  CITT  models.  The  team 
completed  these  activities/outcomes  within  the  constraints  specified  in  the  SOW  indicating  that 
the  CUT  was  to  be  designed  as  a  fully-integrated  training  system  that  attends  to  the  needs  of  the 
senior  commander,  commander,  unit  trainer,  and  training  analyst  as  well  as  the  casual  user. 

Three  specific  design  requirements  were  identified  for  the  CITT.  First,  the  CITT  was  to 
be  designed  to  be  an  information  system.  Second,  it  was  to  be  designed  as  a  folly  integrated 
training  tool  providing  the  user  with  a  means  to  obtain  training  information  from  a  variety  of 
sources.  Finally,  it  was  to  be  designed  to  provide  the  user  with  information  concerning  training 
capabilities  and  a  road  map  for  the  successful  planning,  preparation,  execution,  and  sustainment 
of  training  using  a  structured,  simulation-based  approach  in  CCTT. 

Throughout  the  remainder  of  this  document,  references  to  the  CITT  “system”  refer  to  the 
hardware,  software,  and  documentation  components  that  were  developed  in  the  project  and  that 
comprise  the  CITT.  Specific  discussion  of  hardware  configurations  recommended  for  the  CITT 
system  are  clearly  delineated  within  this  document,  as  are  those  for  the  application  that  is 
demonstrated  in  the  CITT  prototype. 

Needs  Assessment  and  Requirements  Definition 

The  design  of  the  CITT  incorporates  the  widely  accepted  principles  described  in  the 
Methodology  for  the  Development  of  Structured  Simulation-based  Training  (Campbell, 
Campbell,  Sanders,  Flynn,  and  Myers,  1995)  and  is  intended  to  support  training  plans  identified 
by  the  Force  XXI,  Warfighter  XXI,  and  Army  Training  XXI  training  and  campaign  plans.  The 
intent  of  the  process  was  to  produce  a  viable  design  that  would  serve  as  the  basis  for  future 
development  as  well  as  for  the  development  of  a  “proof-of-concepf  ’  prototype.  In  actuality,  two 
designs  were  produced:  the  CITT  ‘To-Be”  design  which  lays  out  the  architecture  and  data 
requirements  for  a  system  which  satisfies  all  of  the  requirements  in  the  SOW;  and  a  CITT  “As- 
Is”  design  which  is  limited  to  a  description  of  the  architecture  and  data  requirements  for  the 
prototype  CITT  actually  built.  According  to  plan,  an  experienced  team  of  training  development, 
instructional  design,  and  simulation  systems  subject  matter  experts,  as  opposed  to  software 
developers,  undertook  the  task  of  designing  the  CITT. 

Identifying  Design  Requirements 

Within  the  context  of  the  above,  the  project  team  completed  an  assessment  of  the  CITT 
system  requirements.  They  envisioned  that  the  CITT  would  provide  the  following: 

1 .  An  introduction  to  the  CCTT  and  its  capabilities  and  limitations. 

2.  An  introduction  to  the  CITT  and  its  capabilities  and  limitations. 

3 .  Access  to  existing  CCTT  TSPs. 

4.  Instructions  and  tools  for  the  use  of  these  TSPs. 

5.  Instructions  and  tools  for  modifying  these  TSPs. 

6.  Instructions  and  tools  for  creating  new  TSPs. 
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7.  Instructions  and  tools  for  building  exercise  files  based  upon  these  TSPs. 

8.  Guidelines  for  testing  or  proofing  TSPs. 

9.  Information  concerning  other  uses  of  CCTT  (e.g.,  gunnery  training,  Situational  Training 
Exercises  (STXs),  etc.). 

10.  Links  to  previously  developed  electronic  aides  and  computer-based  instruction,  such  as 
EDUCCATT  and  demonstrations  of  performance. 

11.  Help  files. 

12.  Detailed  listing  of  Point  Of  Contacts  (POCs)  available  for  CCTT  users. 

In  addition,  there  was  an  implied  requirement  to  link  to  ATIMP  Systems  (e.g.,  SATS  and 
ASAT).  This  link  was  intended  to  ensure  that  information  deemed  essential  to  the  operation  of 
the  tactical  unit  (e.g.,  Mission  Training  Plans,  resource  listings,  and  doctrinal  publications)  was 
available  to  the  CITT  user. 

The  intent  was  to  design  a  system  that  would  satisfy  these  requirements  through  both  a 
standalone  system  and  from  distributed  (Internet)  access.  This  is  consistent  with  the  essence  of 
the  Army  Training  XXI  (AT  XXI)  Campaign  Plan  in  that  it  exploits,  or  directs  users  to  exploit, 
state-of-the-art  technologies  in  information  systems  to  allow  units  to  better  plan,  prepare, 
execute,  and  manage  collective  training  (Department  of  the  Army,  1997b). 

Further,  the  team  envisioned  that  the  CITT  would  be  usable  on  the  personal  computers 
found  within  a  typical  Army  unit’s  Orderly  or  Training  Rooms,  and  also,  that  users  would  want 
to  access  the  CITT  from  the  personal  computer  found  in  their  homes.  The  team  intended  to 
provide  the  design  for  a  robust  system  that  would  last  well  into  the  21st  century. 

In  responding  to  these  needs,  the  team  envisioned  a  single  design  with  two  primary 
components,  the  first  of  which  would  be  an  information  repository  complete  with  information 
concerning: 

1.  The  CITT  itself. 

2.  The  CCTT  system. 

3.  Training  on  the  CCTT. 

4.  The  structured  simulation-based  methodology  developed  using  the  SAT  Process  and 
developed  during  five  plus  years  of  support  to  TSP  development. 

This  information  repository  would  be  based  on  the  10  content  as  described  above. 

The  second  component  would  be  a  fully  interactive  system  allowing  the  user  to  review 
and  modify  existing  TSP  materials  as  well  as  to  create  new  TSPs  in  accordance  with  the 
structured  training  methodology  previously  discussed.  The  design  would  also  provide  links  to 
other  information  systems  that  expand  upon  the  basic  capabilities  of  the  C1'1T  as  described 
above. 


Finally,  the  team  determined  that  the  basic  CITT  software  package  had  to  be  a  complete 
application  and  include  commercial-off-the-shelf  (COTS)  software  capabilities  and  links.  This 
includes  word  processing,  spreadsheets,  a  graphics  package,  and  a  database  repository. 
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Identifying  a  Design  Methodology 


In  assessing  how  to  design,  document,  and  build  an  application  that  satisfies  the  system 
requirements,  the  project  team  conducted  a  needs  analysis  to  determine  how  best  to  proceed. 

This  analysis  revealed  two  primary  concerns.  First,  the  team  recognized  a  need  to  modify  the 
structured  simulation-based  methodology  described  earlier  and  apply  it  to  a  computer-based 
application.  This  training  development  methodology,  as  identified  in  the  various  research  and 
development  programs  sponsored  by  ARI,  is  based  upon  the  Instructional  Systems  Design  (ISD) 
process.  This  process  has  been  codified  as  the  SAT  process  as  identified  in  TRADOC 
Regulation  350-70  (Department  of  the  Army,  1995)  and  is  considered  the  Army’s  training 
development  process.  Second,  because  of  the  possible  integration  of  the  CITT  system  with  other 
ATIMP  Systems,  the  team  determined  a  need  to  adhere  to  Department  of  Defense  (DoD)  and 
Department  of  the  Army  (DA)  regulations  as  well  as  other  guidance  concerning  the  design, 
development,  and  documentation  of  such  systems.  These  regulations  include  DA  Pamphlet  25-1 
(Army  Information  Architecture),  TRADOC  Pamphlet  25-71  (Standards  for  Electronic  Staffing, 
Publication,  and  Archiving,  Department  of  the  Army,  1997d)),  MIL-STD-498  (Software 
Development  and  Documentation),  and  the  Federal  Information  Processing  Publication  183, 
Integration  Definition  for  Function  Modeling  (IDEF)  (U.  S.  Department  of  Commerce,  1993). 

The  first  concern  was  answered  by  adhering  to  the  SAT  process.  This  process  establishes 
a  well-defined  methodology  that  is  applicable  to  both  training  and  the  design  of  training  tools. 
The  steps  included  in  this  process  for  the  design  of  the  CITT  system  are: 

1.  An  initial  design  phase  that  includes  an  assessment  phase  to  determine  requirements,  an 
analysis  phase  that  considers  what  was  learned  from  the  assessment,  and  a  design  phase 
where  the  results  of  the  analysis  serve  as  a  baseline. 

2.  A  development  phase  intended  to  produce  the  design. 

3.  A  formative  evaluation  phase  (for  both  design  and  development  phases). 

4.  An  implementation  phase  where  recommendations  for  fielding  the  developed  product  are 
made. 

The  use  of  the  SAT  process  required  modification  by  the  project  team  for  two  reasons. 
First,  the  project  team  was  operating  on  a  tight  schedule  which  required  shortening  some 
activities.  Second,  team  members  were  acting  as  both  designer/developers  as  well  as  subject 
matter  experts  on  the  original  methodology. 

The  second  concern  was  addressed  by  the  project  team’s  attention  to  DoD  and  DA 
guidance  in  documenting  the  design  of  the  CITT  application,  particularly  as  related  to  the 
potential  integration  of  the  CITT  application  into  ATIMP  Systems.  The  team  initiated  a  close 
investigation  of  DA  policies  concerning  the  design  and  development  of  software  applications. 
This  investigation  identified  ATSC,  a  subordinate  command  of  TRADOC,  as  the  proponent 
agency.  This  lead  to  a  significant  finding  concerning  software  solutions  that  would  facilitate  the 
design  process.  The  project  team  reviewed  the  SAT,  ASAT,  and  Combined  Arms  Training 
Strategy  (CATS)  Systems  and  discovered  that  all  were  initially  designed  and  developed  using 
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common  business  process  reengineering  applications  and  tools6.  To  attend  to  the  challenges 
inherent  in  enterprise  engineering,  the  DoD  and  DA  have  begun  to  use  Integrated  Definition 
modeling. 

The  Integrated  DEFinition  (IDEF)  methodology  is  a  suite  or  family  of  methods  that 
supports  the  modeling  needs  of  an  enterprise  and  its  business  areas.  IDEF  technologies  have 
grown  over  the  past  several  years  and  are  used  widely  by  both  the  DoD  and  some  of  the  largest 
U.S.  corporations.  Although  IDEF  was  originally  intended  for  use  in  systems  engineering,  the 
suite  of  IDEF  methods  has  evolved  and  contains  the  necessary  notations  to  support  software 
development.  When  constructed  properly,  IDEF0,  EDEF1,  IDEF IX,  and  IDEF3  models  can  be 
complementary  to  software  engineering  business  rules.  Their  uses  are  described  below  (Mayer, 
Benjamin,  Caraway  and  Painter,  1998): 

1.  IDEF0  modeling  is  used  to  produce  a  function  or  activity  model  that  is  a  structured 
representation  of  the  functions  of  the  system  or  environment  and  of  the  information  and 
objects  that  interrelate  those  functions  or  activities. 

2.  IDEF1  modeling  is  used  to  produce  an  information  model  that  represents  the  structure  of 
information  needed  to  support  the  functions  or  activities  of  the  system  or  environment. 

3.  EDEF1X  modeling  is  used  for  designing  relational  database  schema  of  the  system  under 
development. 

4.  IDEF3  modeling  is  used  to  capture  descriptions  of  sequences  of  activities.  The  primary  goal 
of  EDEF3  is  to  provide  a  structured  method  by  which  a  subject  matter  expert  can  express 
knowledge  about  the  operation  of  a  particular  system  or  organization. 

IDEF  began  when  the  U.  S.  Air  Force,  in  response  to  the  identification  of  the  need  to 
improve  manufacturing  operations,  established  the  Integrated  Computer-Aided  Manufacturing 
(ICAM)  program  in  the  mid-1970s.  A  major  development  from  the  ICAM  program  was  the 
Integrated  DEFinition  methodology.  This  methodology  was  to  be  used  as  a  regimented  approach 
to  analyzing  an  enterprise,  capturing  "as-is"  process  models,  (i.e.,  models  of  the  system  as  it 
currently  exists),  and  for  modeling  activities  (organizational  units)  within  an  enterprise.  As 
noted  previously,  these  methodologies  have  evolved  and  contain  the  necessary  notations  to 
support  software  development  and  as  such  are  in  full  use  by  ATSC7. 

The  Design  Process 

Design  Considerations 

The  modeling  tools  identified  above  created  the  basic  design  for  the  CITT.  The  key  to 
the  initial  design  of  the  CITT  was  in  deriving  user  and  system  requirements,  identifying  key 


6  The  term  “business  process  reengineering”  has  been  used  primarily  in  the  context  of  the 
redesign  of  industry  as  it  restructures  to  meet  evolving  requirements  and  new  industry  demands 
while  remaining  competitive.  This  usage  has  not  been  employed  within  the  DoD  and  DA 
spheres  of  influence.  Within  DoD,  the  accepted  term  is  “enterprise  engineering”  which 
represents  a  structured  approach  to  adapting  past  practices  to  new  requirements. 

7  The  requirement  for  using  this  methodology  comes  from  the  DoD  Enterprise  Model, 
[Department  of  Defense,  1993]  and  has  been  reinforced  by  recent  guidance  from  ATSC. 
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issues  and  functions/activities  as  identified  earlier  in  this  section,  determining  interfaces,  and 
allocating  architecture  requirements  to  system  elements.  This  involved  combining  the  SAT 
process  and  IDEF  modeling.  The  team  used  an  iterative  design  process  completed  in  two  forms 
to  determine  requirements.  The  products  of  this  process  included  an  activity  model  and  a 
process  model.  The  project  team  elected  to  use  Commercial-off-the-Shelf  (COTS)  software, 
specifically  BPwin®8  to  support  all  activity  modeling  and  system  design.  This  selection  ensured 
compliance  with  known  ATSC  usage  as  well  as  facilitated  the  development  of  the  CITT 
prototype. 

The  team  used  BPwin  to  detail  the  model  to  include  all  known  activities  that  comprise  the 
overall  system  design.  Outputs  from  this  modeling  process  include  diagrams  and  activities  and 
form  the  basis  for  IDEF0  modeling.  BDEF0  was  useful  in  establishing  the  scope  of  the  analysis 
to  determine  user  needs  especially  for  a  functional  analysis.  As  a  communication  tool,  DDEF0 
enhanced  subject  matter  expert  involvement  through  simplified  graphical  devices.  As  an 
analysis  tool,  EDEF0  assisted  in  identifying  functions  to  be  performed  and  activities  needed  to 
perform  those  functions. 

The  Design  Process  in  Macro  Form 

The  basic  IDEF0  concepts  used  in  the  design  of  the  CITT  included  the  following. 
(Mayer,  et  al.,  1998): 

Cell  Modeling  Graphic  Representation.  The  "box  and  arrow"  graphics  of  an  IDEF0 
diagram  show  the  activity  or  function  as  a  box  and  the  interfaces  to  or  from  the  function  as 
arrows  entering  or  leaving  the  box.  These  interfaces  (the  Inputs,  Controls,  Outputs,  and 
Mechanisms  or  ICOMs)  serve  a  specific  function  in  helping  to  describe  the  model. 

1 .  Inputs  represent  materials  or  information  transformed  or  consumed  by  the  activity. 

2.  Controls  represent  constraints  on  the  activity  with  respect  to  how,  when,  and/or  if  an  activity 
is  performed.  Controls  are  not  transformed  as  a  result  of  the  activity. 

3.  Outputs  represent  materials  or  information  that  were  produced  by  the  activity. 

4.  Mechanisms  represent  a  person,  machine,  or  other  non-consumable  resource  used  to  perform 
the  activity. 

To  express  functions,  boxes  operate  simultaneously  with  other  boxes  with  the  interface  arrows 
"constraining"  when  and  how  operations  are  triggered  and  controlled.  A  basic  model  Graphic 
Representation  is  depicted  in  Figure  6. 

Communication.  IDEF0  concepts  designed  to  enhance  communication  included  the 
following: 

1.  Diagrams  based  on  simple  box  and  arrow  graphics  with  text  labels  to  describe  boxes  and 
arrows,  and  glossary  and  text  to  define  the  precise  meanings  of  diagram  elements. 

2.  The  gradual  exposition  of  detail  featuring  a  hierarchical  structure  with  the  major  functions  at 
the  top  and  with  successive  levels  of  sub-functions  revealing  breakouts  of  well  bounded 
details. 


8  BPWin  is  a  registered  trademark  of  PLATINUM  Technology,  Inc. 
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3.  A  "node  tree”  or  “node  chart"  that  provides  a  quick  index  for  locating  details  within  the 
hierarchical  structure  of  diagrams.  A  basic  node  tree  is  depicted  at  Figure  7. 

4.  The  limitation  of  detail  to  no  more  than  seven  sub-functions  on  each  successive  function. 


Control 


Input 


Output 


Figure  6.  An  example  of  a  Graphic  Representation. 

Rigor  and  Precision.  The  rules  of  EDEF0  require  sufficient  rigor  and  precision  to  satisfy 
the  designers’  needs  without  overly  constraining  the  analysis  conducted  in  the  design  of  any 
system.  As  such,  IDEF0  rules  provide  the  following: 

1 .  Control  of  the  details  communicated  at  each  level  (three  to  seven  function  boxes  at  each  level 
of  decomposition). 

2.  Bounded  Context  (no  omissions  or  additional  out-of-scope  detail). 

3.  Diagram  Interface  Connectivity  (Node  numbers,  Box  numbers,  chronological  creation 
number  or  C-number,  and  Detail  Reference  Expression). 

4.  Data  Structure  Connectivity  often  referred  to  as  ICOM  codes.  As  described  above,  ICOM  is 
an  acronym  for  Inputs,  Controls,  Outputs,  and  Mechanisms  and  is  best  identified  as  the 
arrows  found  in  “For-Exposition-Only”  (FEO)  diagrams. 

5.  Unique  Labels  and  Titles  (no  duplicated  names). 

6.  Syntax  Rules  for  Graphics  (boxes  and  arrows). 

7.  Data  Arrow  Branch  Constraints  (labels  for  constraining  the  data  flow  on  branches). 

8.  Input  versus  Control  Separation  (a  rule  for  determining  the  role  of  data). 

9.  Data  Arrow  Label  Requirements  (minimum  labeling  rules). 

10.  Minimum  Control  of  Function  (all  functions  require  at  least  one  control). 

11.  Purpose  and  Viewpoint  (all  models  have  a  purpose  and  viewpoint  statement). 

Methodology.  Step-by-step  procedures  are  provided  for  modeling,  review,  and 
integration  tasks. 

Organization  versus  Function.  The  separation  of  organization  from  the  function  was 
included  by  the  selection  of  functions  and  interface  names  during  the  development  of  the  model. 

Sequence  and  Timing  Independence.  Applying  the  IDEF0  method  resulted  in  an 
organized  representation  of  the  activities  and  the  important  relations  between  these  activities. 
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•  Activity  •  Activity  •  Activity  •  Activity  •  Activity 

•  Activity  •  Activity  •  Activity  •  Activity  •  Activity 


Figure  7.  An  example  of  a  Node  Tree. 

Strengths  and  Weaknesses  of  IDEF0  Modeling 

The  primary  strength  of  using  IDEF0  modeling  was  that  it  proved  effective  in  detailing 
the  CITT  activities  for  the  function  model.  Additionally,  the  description  of  the  activities  of  the 
CITT  were  easily  refined  into  greater  and  greater  detail  until  the  model  proved  to  be  as 
descriptive  as  necessary  for  decision-making.  In  fact,  one  of  the  observed  problems  with  IDEF0 
modeling  of  the  CITT  was  that  it  was  overly  concise.  In  reviewing  existing  IDEF0  models  from 
ATSC,  specifically  SATS,  ASAT,  and  CATS,  the  project  team  found  that  these  models  were  so 
concise  that  they  were  understandable  only  if  the  reader  was  an  expert  in  the  described  system  or 
had  participated  in  the  model  development.  While  this  initially  posed  a  challenge  to  the  team,  a 
review  of  those  systems  helped  to  clarify  design  and  function  questions  for  the  CITT  system. 
This  provided  the  project  team  with  direction  in  both  the  selection  and  use  of  BPwin. 

Once  decisions  had  been  made  regarding  the  modeling  process  and  tools  that  would  be 
used  to  design  and  describe  the  CITT,  the  actual  modeling  process  began.  While  modeling  was 
substantially  based  on  the  assessment  activities  that  occurred  early  in  the  project,  of  equal 
importance  was  the  combined  subject  matter  expertise  of  the  project  team  in  structured  training 
principles  and  concepts  and  in  the  operation  of  the  CCTT.  Also,  since  the  CITT  was  an  entirely 
new  system,  an  analytical  modeling  process  had  to  be  developed.  That  is,  because  there  was  no 
existing  system  that  could  be  “disassembled”  to  describe  and  model  it,  the  team  had  to  model  as 
a  sequence  of  “what  if’  analyses.  They  also  had  to  determine  from  whose  viewpoint  modeling 
would  occur.  The  decision  was  made  to  model  from  the  viewpoint  of  the  unit  commander/unit 
trainer. 
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The  result  of  IDEF  modeling  was  a  complete  exposition  of  the  CITT  which  met  all 
requirements  for  IDEF0  modeling  and  is  discussed  below  under  CITT  “To-Be”  Design. 
Unfortunately,  the  model  proved  to  be  insufficient  in  attending  to  the  needs  of  the  prototype 
developers  due  in  large  part  to  the  fact  that  even  though  the  designers  clearly  knew  the  desired 
functionality  for  the  CITT,  the  IDEF0  model  did  not  document  items  that  are  important  in  the 
actual  development  of  a  system.  Such  items  include  actual  graphical  user  interfaces  (GUI)  and 
desired  functionality  within  the  model.  This  produced  some  difficulties  for  initial  prototype 
development.  When  the  results  of  activity  modeling  were  given  to  the  developers,  there  was  a 
tendency  to  interpret  the  model  as  representing  a  sequence  of  activities,  and  even  though  the 
project  developers  understood  that  IDEF0  should  not  be  used  for  modeling  activity  sequences,  it 
was  difficult  not  to  do  so.  It  was  natural  to  order  the  activities  because  if  one  activity’s  outputs 
are  used  as  input  by  another  activity,  drawing  the  activity  boxes  and  concept  connections  is 
clearer.  This  tendency  became  less  problematic  as  modeling  and  prototype  development 
progressed  and  the  team  gained  more  experience. 

A  second  problem  concerned  the  difficulty  the  project  team  had  in  separating  the  design 
of  CITT  from  the  development  of  the  CITT  prototype.  The  need  to  develop  a  prototype  was  a 
given  from  the  start  of  the  project,  however,  it  was  stressed  that  the  primary  purpose  of  the 
project  was  to  design  the  CITT.  Nevertheless,  in  early  modeling  sessions,  there  was  significant 
difficulty  separating  design  from  prototype.  It  was  common  for  design  decisions  to  be 
confounded  by  a  discussion  of  whether  or  not  the  prototype  could  be  made  to  work  as  the  design 
was  specifying.  This  was  further  confounded  by  considerations  based  on  the  specific  software 
that  had  been  selected  for  prototype  development. 

The  project  team  solved  these  difficulties  by  adopting  a  strict  working  process  for 
modeling  which  specified  that  in  the  initial  design  and  modeling  of  an  activity,  no  discussion  was 
allowed  regarding  how  that  activity  would  be  implemented  in  the  prototype.  Only  when 
modeling  for  an  activity  was  completed  was  the  discussion  opened  to  implementation.  This 
meant  that  activities  were  modeled  and  approved  prior  to  discussions  with  the  project  team's 
developers.  Once  completed,  the  designers  briefed  their  model  to  the  project  team  at  which  point 
developers  and  subject  matter  experts  for  training,  military  operations,  and  the  ISD/SAT  process 
began  to  determine  their  individual  requirements  in  light  of  the  approved  design.  Designers, 
developers,  and  SMEs  were  kept  on  track  by  an  active  project  team  management  that  acted  as 
both  a  clearinghouse  and  arbitrator  of  contentious  issues.  Additionally,  this  assisted  the  project 
team  in  clearly  differentiating  between  design  and  prototypical  application  requirements  and  the 
development  of  logical  and  physical  models  that  attended  to  these  requirements.  Because  of  this 
adjustment,  modeling  proceeded  much  more  efficiently. 

The  CITT  Design 

As  could  be  anticipated  when  using  an  analytical  design  process,  modeling  occurred 
hierarchically.  That  is,  the  team  started  with  top-level  activities  and  proceeded  to  decompose 
them  into  their  lower-level  activities  using  the  revised  modeling  process  just  described.  This 
logical  decomposition  process  continued  until  the  desired  level  of  detail  was  achieved.  The 
result  was  the  CITT  ‘To-Be”  model  (i.e.,  the  design  of  the  CITT  as  specified  in  the  SOW 
independent  of  how  that  design  would  be  implemented  in  prototype).  As  modeling  and 
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prototype  development  proceeded,  an  “As-Is”  model  was  also  documented  which  illustrates  the 
CITT  prototype.  The  remainder  of  this  section  will  describe  the  "To-Be”  and  “As-Is”  models. 

The  CITT  “To-Be”  Design 

The  diagram  shown  in  Figure  8,  CITT  -  Context  Diagram,  depicts  the  top-most  level  of 
the  “To-Be”  design  of  the  CITT.  This  diagram  establishes  the  general  bounds  of  the  CITT 
model.  It  also  represents  the  boundary  of  the  process  with  respect  to  purpose,  scope,  and 
viewpoint.  Thus,  none  of  the  decompositions  (or  children)  of  this  parent  diagram  may  include 
factors  not  considered  in  it.  As  the  activity  is  further  decomposed  and  component  functions 
identified,  the  result  is  a  series  of  For  Exposition  Only  (FEO)  diagrams  that  depict  the  design. 


Figure  8.  CITT  -  Context  Diagram. 

The  diagram  shown  in  Figure  9,  CITT  -  Design  Top-Level  FEO,  is  the  first  of  the  FEO 
diagrams  that  resulted  from  modeling.  A  FEO  is  defined  as  a  diagram  that  depicts  two  or  more 
sub-processes  of  an  associated  parent  activity.  It  is  also  referred  to  as  a  child  diagram.  Figure  9 
depicts  the  first  level  of  activities  decomposed  from  the  top-level  diagram  in  Figure  8.  It 
establishes  the  major  activities  or  functions  envisioned  for  the  CITT.  Note  also  that  although  it 
is  graphically  more  complex  than  its  parent,  its  boundaries  remain  consistent  with  those 
established  by  the  Context  Diagram  consistent  with  the  definition  of  an  ICOM  Graphic 
Representation  as  shown  in  Figure  6. 
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Figure  9.  CITT  -  Design  Top-Level  FEO. 

The  graphical  representation  of  the  Context  Diagram  and  all  of  its  associated  children  provides  a 
detailed  design  of  the  CITT  system.  For  the  ‘To-Be”  CITT,  the  total  number  of  FEOs  is 
approximately  forty.  The  complete  diagrammatic  representation  of  the  ‘To-Be”  CITT  is 
contained  in  CUT  Design  (To-Be)  Documentation  (CITT  Team,  1998a). 

For  ease  in  detailing  the  CITT  "To-Be"  model,  a  series  of  node  tree  diagrams  was 
developed  to  facilitate  quick  navigation  through  the  CITT  "To-Be."  These  were  completed  for 
the  project  team  as  well  as  for  individuals  who  may  not  be  familiar  with  the  various  FEOs  that 
are  the  CITT  design.  The  diagram  shown  in  Figure  10  is  a  sample  node  tree  for  the  top-level  and 
clearly  identifies  all  functions  or  activities  contemplated  through  three  levels  of  decomposition. 
To  assist  the  reader  in  understanding  the  CITT  design,  all  of  the  node  tree  diagrams  are  included 
in  Appendix  B. 

To  further  elucidate  the  design,  the  project  team  employed  a  feature  of  BPwin  that 
converts  the  FEO  diagrams  into  text  entries  that  are  dynamically  linked  to  the  object-oriented 
model.  These  text  entries  are  referred  to  as  Activity  Listings,  and  they  provide  descriptive 
information  of  the  activity:  definitions,  inputs,  controls,  outputs,  mechanisms,  and  off-page  and 
external  references.  The  project  team  used  this  simplified  process  with  great  success  in  detailing 
the  design  of  the  CITT  to  approving  authorities.  When  used  in  conjunction  with  the  design 
diagrams,  these  documents  provide  a  total  description  of  the  CITT  system.  The  ‘To-Be” 

Activity  Listings  are  also  included  in  CITT  Design  (To-Be)  Documentation  (CITT  Team, 

1998a). 
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Figure  10.  The  ‘To-Be”  node  tree  diagram. 

The  “To-Be”  CITT  is  a  five-node  system.  For  ease  of  navigation,  a  Node  Tree  diagram  is 
included  in  CITT  Design  (To-Be')  Documentation  tCITT  Team,  1998a).  CITT  is  designed  to 
provide  information  to  the  user  concerning  both  the  CITT  and  CCTT.  Additionally,  the  CITT 
provides  a  simple,  yet  powerful  authoring  tool  for  the  CCTT  user  to  review  and  develop 
exercises  designed  specifically  for  the  CCTT.  The  major  components  of  the  CUT  Design  are: 

Examine  CITT.  This  component  provides  information  concerning  CITT,  its  intended 
use,  development,  capabilities,  and  limitations  as  well  as  detailed  guidance  on  navigation  options 
for  the  user.  It  includes  warnings  concerning  the  difficulty  of  Modify  and  Create  activities. 

Learn  about  CCTT.  This  component  includes  information  concerning  CCTT,  its 
innovative  uses,  development,  capabilities,  and  limitations.  It  includes  up-to-date  CCTT  system- 
specific  information  in  graduated  levels  (to  attend  to  the  needs  of  casual  &  specialized  users)  as 
well  as  information  concerning  how  to  train  using  CCTT  based  upon  task-based,  structured 
methodology.  It  includes  live  links  to  existing  computer-based  training  (CBT)  systems  such  as 
EDUCCATT  and  the  Demonstrations  of  Performance  as  well  as  future  CBT  systems. 

Pmdnnft  Training  Materials.  This  component  includes  a  "how  to"  prepare  and  use 
structured  TSPs  based  upon  the  structured,  simulation-based  methodology.  It  provides 
information  for  reviewing  the  structured  training  exercises  within  CCTT,  and  how  to  select, 
modify,  or  create  a  CCTT  exercise.  This  is  the  most  complex  component  of  the  system  and 
includes  links  to  sources  of  information  concerning  weapons  systems,  doctrine,  and  tactics  as 
well  as  to  all  archived  exercises  developed  in  and  subsequent  to  the  initial  STRUCCTT  and 
STRUCCTT-2  efforts.  Additionally,  it  includes  an  electronic  rendering  of  the  terrain  databases 
upon  which  users  will  conduct  their  exercises  once  in  the  CCTT.  This  rendering  is  expected  to 
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be  as  accurate  as  the  plan  view  displays  found  in  the  CCTT  and  will  include  the  capability  to 
“wargame”  an  exercise  even  before  it  is  loaded  into  the  CCTT.  Finally,  this  component  includes 
extensive  tutorial/help  features  based  upon  lessons  learned  by  various  contract  teams  since  the 
inception  of  the  Virtual  Training  Program. 

Produce  Exercise  Files.  This  component  provides  the  means  by  which  the  user  can 
modify  or  create  a  CCTT  exercise  file  based  upon  materials  identified  or  developed  in  the 
Produce  Training  Materials  section.  It  includes  links  to  documents  created  in  the  previous 
section,  such  as  Exercise  Plansheets,  Event  Guides,  Graphic  Control  Measure  (GCM)  data,  and 
Executable  Overlay(s)  data  for  an  exercise.  In  the  most  advanced  design  form,  it  is  envisioned 
that  this  capability  will  allow  for  the  direct  import  of  exercise  data,  via  standardized  data  tables, 
to  the  CCTT  system.  Transmission  will  be  accomplished  either  electronically  (e.g.,  file  transfer 
protocol)  or  via  3.5”  disk  or  tape  archive.  It  includes  a  tutorial/help  feature. 

Execute  Support  Functions.  This  component  provides  information  and  general 
"housekeeping"  functions  internal  to  the  application.  It  includes  the  actual  links  to  other  source 
material  identified  above  and  is  considered  a  portion  of  the  system  itself.  It  includes  a 
tutorial/help  feature. 

While  general  in  nature,  the  descriptions  above  provide  an  overview  of  the  CITT  design 
which  was  the  primary  objective  of  this  project.  When  the  complete  CITT  design  as  contained  in 
CITT  Design  (To-Bet  Documentation  (CITT  Team,  1998a)  is  examined,  its  detail  is  sufficient 
enough  to  allow  the  experienced  application  developer  working  with  the  designer  to  develop  a 
robust  system  that  attends  to  all  design  parameters.  In  many  cases,  the  design  may  actually  be 
more  complete  than  required  by  the  developer.  If  an  electronic  version  of  the  model  is  available, 
such  as  is  the  case  in  BPwin,  the  basic  design  can  be  imported  into  a  relational  database  design 
system  (such  as  ERwin®9)  for  actual  development. 

The  CITT  “As-Is”  Design 

The  diagram  represented  in  Figure  1 1,  CITT  -  Prototype  Context  Diagram,  depicts  the 
“As-Is”  version  of  the  CITT.  The  “As-Is”  design  is  very  similar  to  the  ‘To-Be”  design,  as  would 
be  expected,  however,  it  reflects  only  those  functionalities  actually  built  into  the  prototype.  This 
diagram  is  the  highest  level  for  the  “As-Is”  model.  It  should  be  noted  that  the  CITT  design  effort 
for  the  prototype  was  conducted  nearly  concurrently  with  that  of  the  projected  design  or  ‘To-Be” 
model.  Note  also  that  the  general  bounds  for  the  model  remain  similar  to  those  identified  in  the 
‘To-Be”  model.  No  attempt  is  being  made  to  compare  the  “As-Is”  to  the  “To-Be”  model;  rather, 
the  “As-Is”  model  is  presented  for  completeness  and  in  keeping  with  standard  industry  modeling 
practices. 

The  diagram  shown  in  Figure  12,  CUT  -  Prototype  Top-Level  FEO,  is  the  initial  FEO 
for  the  “As-Is”  model.  Note  that  it  differs  from  Figure  9,  CITT  -  Design  Top-Level  FEO,  in  the 
number  of  major  modules  found  within  the  general  model.  This  reflects  the  moment  in  time 
when  the  prototype  model  was  “frozen”  while  the  development  of  the  design  version  continued. 


9  ERwin  is  a  registered  trademark  of  PLATINUM  Technology,  Inc. 
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Figure  1 1.  CUT  -  Prototype  Context  Diagram. 


Figure  12.  CUT  -  Prototype  Top-Level  FEO. 
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The  CITT  model  for  the  prototype  as  well  as  a  Node  Tree  diagram  are  included  in  the 
CITT  Prototype  { As-Is)  Documentation  (CITT  Team,  1998b.)  The  major  components  of  the 
Cm  prototype  design  are: 

Examine  CITT.  This  component  includes  information  concerning  CITT,  its  intended  use, 
development,  capabilities,  and  limitations  for  the  user.  It  includes  general  information  about  the 
system.  It  includes  warnings  concerning  the  difficulty  of  Modify  and  Create  activities. 

How  to  Navigate  CITT.  This  component  provides  information  concerning  navigating  the 
CITT  system  by  role  (e.g.,  as  a  unit  trainer,  participant  in  training,  etc.)  or  by  needs  (e.g.,  to 
obtain  a  specific  piece  of  information  concerning  a  specific  function). 

Learn  about  CCTT.  This  component  includes  information  concerning  CCTT,  its 
innovative  uses,  development,  capabilities,  and  limitations.  It  includes  known  CCTT  system- 
specific  information  in  graduated  levels  (to  attend  to  the  needs  of  casual  and  specialized  users)  as 
well  as  information  concerning  how  to  train  using  CCTT  based  upon  task-based,  structured 
methodology.  It  includes  information  concerning  other  computer-based  training  (CBT)  systems 
such  as  EDUCCATT  and  the  Demonstrations  of  Performance  and  directs  the  user  to  systems 
where  these  resources  can  be  accessed. 

Produce  Training  Materials.  This  component  includes  a  "how  to"  prepare  and  use 
structured  TSPs  based  upon  the  structured,  simulation-based  methodology.  It  provides 
information  for  reviewing  the  structured  training  exercises  within  CCTT,  and  how  to  select, 
modify,  or  create  a  CCTT  exercise.  This  is  the  most  complex  section  of  the  system.  It  includes 
information  concerning  weapons  systems,  doctrine,  and  tactics  and  where  this  information  can  be 
found.  It  also  provides  access  to  exercises  developed  under  government  contract  that  were 
available  at  the  initiation  of  the  CITT  effort.  Additionally,  users  receive  an  electronic  rendering 
of  the  terrain  database  upon  which  they  will  conduct  their  exercise  once  in  the  CCTT.  This 
rendering  is  nearly  as  accurate  as  the  plan  view  displays  found  in  the  CCTT.  It  includes 
extensive  tutorial/help  based  upon  lessons  learned  by  various  contract  teams  since  the  inception 
of  the  Virtual  Training  Program. 

Produce  Exercise  Files.  This  component  provides  the  means  by  which  the  user  can  learn 
how  a  typical  CCTT  site  modifies  or  creates  a  CCTT  exercise  file  based  upon  materials 
identified  or  developed  in  the  Produce  Training  Materials  section. 

Exercise  Management  Tools.  This  component  provides  a  “quick  look-up”  capability  that 
allows  users  to  check  on  the  status  of  an  exercise  file  that  they  may  have  authored.  It  is  used  in 
conjunction  with  the  “View”  and  “Modify”  options  found  under  the  Produce  Training  Materials 
component 

Execute  Support  Functions.  This  component  provides  information  and  general 
"housekeeping"  functions  internal  to  the  application.  It  includes  existing  hyperlinks  and  other 
navigational  aids  that  are  found  within  the  prototypical  CITT  and  is  considered  a  portion  of  the 
system  itself.  It  includes  a  tutorial/help  feature. 

As  with  the  "To-Be"  application,  the  team  developed  a  series  of  node  tree  diagrams  for 
the  "As-Is"  application  to  facilitate  quick  navigation  through  the  CITT  prototype.  Again,  these 
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were  completed  for  the  project  team  as  well  as  for  those  not  necessarily  familiar  with  the  various 
FEOs  that  are  the  CITT  design.  The  diagram  shown  in  Figure  13  is  a  sample  node  tree  for  the 
top-level  and  clearly  identifies  all  functions  or  activities  contemplated  through  three  levels  of 
decomposition. 


Figure  13.  The  “As-Is”  node  tree  diagram. 


When  the  descriptions  above  are  compared  with  those  of  the  ‘To-Be”  model  (Figure  10), 
a  striking  similarity  can  be  noted.  This  is  because  the  “As-Is”  model  is  simply  a  lesser  version  of 
the  “To-Be”  model.  Because  the  project  team  had  the  luxury  of  designing  for  both  ‘To-Be”  and 
“As-Is”  simultaneously,  this  was  a  conscious  design  decision.  Simply  stated,  the  “As-Is”  CITT 
is  a  simplified  rendering  of  the  design  presented  in  the  ‘To-Be”  CliT  based  upon  that  portion  of 
the  “To-Be”  design  that  was  achievable  in  the  CITT  prototype. 

The  final  step  required  in  the  design  of  the  CITT  system  prototype  was  the  specification 
of  hardware  and  software  requirements.  In  reviewing  this  requirement,  the  project  team 
attempted  to  use  both  industry  and  emerging  standards  to  identify  an  appropriate  hardware  suite 
for  each  prototype.  An  implied  task  was  to  identify  a  target  system  that  met  requirements 
identified  by  ATSC  for  use  with  evolving  ATIMP  Systems  (e.g.,  SAT,  ASAT,  and  CATS).  The 
only  solution  appeared  to  be  that  which  the  United  States  Military  Academy  at  West  Point  uses 
as  its  common  "desktop"  for  Corps  of  Cadet  members.  Colloquially  referred  to  as  the  "West 
Point  Standard,"  this  solution,  identified  in  Table  1,  provides  for  longevity  as  well  as 
affordability.  It  is  important  to  note  that  although  these  solutions  (identical  except  for  the 
Internet  access  requirement)  are  considered  "low-end"  systems  by  current  industry  standards, 
they  exceed  the  current  capabilities  of  most  systems  found  in  Directorate  of  Information 
Management  (DOIM)  fielded  sites.  However,  these  systems  can  be  obtained  by  DOIMs  or  their 
authorized  representatives  at  minimal  cost — currently  less  than  $2000  per  system. 
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Table  1.  Prototype  Hardware/Software  Specifications. 


Hardware 

Standalone  Prototype 

Distributed  Prototype 

Chip  Set 

Pentium  233 

Pentium  233 

Random  Access 
Memory 

64  Megabyte 

64  Megabyte 

Hard  Drive 

2  Gigabyte 

2  Gigabyte 

Compact  Disk-Read 
Only  Memory 

Multimedia  PC  Computing 

Specification  3 

Multimedia  PC  Computing 

Specification  3 

Video  Card 

Video  Graphics  Adapter  Card  with  4 
Megabyte  Video  Random  Access 

Memory 

Video  Graphics  Adapter  Card  with  4 
Megabyte  Video  Random  Access 

Memory 

Sound  Card 

Sound  Blaster  Compatible  with  Speakers 

Sound  Blaster  Compatible  with  Speakers 

Modem 

Not  applicable 

28.8  Kilobits  per  second 

Screen  Display 

17”  Monitor,  800  x  600  resolution 

17”  Monitor,  800  x  600  resolution 

Printer  Driver 

Hewlett-Packard  Laser  Jet  4L  Driver 

Hewlett-Packard  Laser  Jet  4L  Driver 

Operating  System 

NT  Workstation  4.0  w /  Service  Pack  3 

NT  Workstation  4.0  w/  Service  Pack  3 

Browser 

Customized  CITT  Browser 

Microsoft  Internet  Explorer  4.01 

Other  Applications 

Microsoft  Office  97  Professional 

Microsoft  Office  97  Professional 

Task  4:  Develop  a  Prototype  CITT  and  Refine  it  Through  Formative  Evaluation 

The  SOW  specified  the  development  and  evaluation  of  a  prototype  CITT  in  standalone 
and  distributed  (World  Wide  Web)  versions.  The  prototype  was  to  include:  complete 
development  of  the  instructional  overview  as  both  a  standalone  (videotape)  and  as  an 
introductory  component  of  the  CITT;  information  needed  on  the  execution  of  structured  training 
with  the  CCTT;  and  links  (at  least  conceptual)  to  existing  bodies  of  information  (such  as 
EDUCCATT)  that  need  to  be  accessed  from  the  CITT.  The  prototype  CITT  was  to  provide 
commanders  and  other  unit  trainers  with  the  capabilities  to  select,  modify,  and  develop  CCTT 
exercises  for  platoons  and  company  teams,  to  access,  modify,  develop  and  print  required  training 
support  materials,  and  to  execute  folly  the  exercises  selected,  modified,  or  developed.  A 
formative  evaluation  of  the  prototype  CITT  was  to  be  conducted  by  monitoring  its  use  by 
commanders  and  other  unit  trainers.  This  section  of  the  report  describes  the  development  of  the 
prototype  CITT.  Formative  evaluation  will  be  described  later  under  project  evaluation. 

For  additional  information  on  the  development  and  structure  of  the  CITTS  A,  see  the 
System  Administrator’s  Manual  for  the  CITT  Prototype  (Standalone  Version)  (CITT  Team, 
1998e)  and  the  Programmer’s  Manual  for  the  CITT  Prototype  (Standalone  Version)  (CITT 
Team,  1998c).  For  additional  information  on  the  development  and  structure  of  the  CITTDT,  see 
the  System  Administrator  &  Programmer’s  Manual  for  the  CITT  Prototype  (Distributed 
Version).  (CITT  Team,  1998d). 
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Prior  to  describing  Task  4,  a  clarifying  note  is  in  order.  Although  this  report  separates  the 
tasks  as  if  they  are  independent,  there  was  a  great  deal  of  overlap  between  Tasks  3  and  4  which 
had  been  anticipated.  As  modeling  was  occurring,  particularly  for  the  “As-Is”  model,  the 
development  of  the  prototype  was  being  done  simultaneously.  Discussion  was  occurring  and 
decisions  were  being  made  that  impacted  both  design  and  prototype  all  within  the  context  of  the 
same  work  session.  Thus,  while  the  two  tasks  can  be  separated  conceptually,  the  reader  should 
bear  in  mind  that  in  design  and  prototype  development,  there  was  often  little  or  no  separation,  at 
least  in  the  early  phases  of  Task  4. 


Development  Approach 

Prototype  development  began  with  requirements  identification  in  which  the  project  team 
collected  and  analyzed  data  on  the  prototype  CITT  functions  and  activities  as  they  would  be 
implemented  in  an  actual  working  system.  While  the  functions  and  activities  the  CITT  prototype 
would  perform  were  being  identified  primarily  from  the  design  activities  occurring  in  Task  3,  the 
team  still  needed  to  determine  how  these  could  best  be  presented  to  the  CITT  user.  To  satisfy 
this  need,  a  User  Jury  consisting  of  a  group  of  key  end  users  representative  of  the  CITT  user 
population  was  identified  and  was  used  in  making  a  number  of  design  decisions.  The  purpose  of 
the  jury  was,  as  McConnell  (1998)  states,  for  the  project  team  to  “ask  the  users  what  they  want, 
show  them  what  they  intend  to  build,  and  ask  them  how  they  like  it — then  listen  carefully  until 
they  fully  understand  both  the  stated  and  unstated  elements  of  the  users’  responses.”  The  jury 
proved  very  important  in  assisting  the  development  team  at  this  stage  of  the  prototype 
development;  as  will  be  discussed  below,  they  also  proved  very  valuable  in  the  evaluation  of  the 
prototype. 

Shortly  after  work  on  Task  4  began,  the  team  built  a  User  Interface  Prototype  (UEP).  A 
UIP  consists  of  proposed  mock-ups  of  screens  for  the  software  system  under  development  that  is 
created  for  the  purpose  of  eliciting  user  feedback  about  the  software’s  intended  functionality  and 
look  and  feel.  The  team  employed  various  alternatives  for  displaying  and  using  CITT  functions 
in  building  the  UIP.  Initially,  the  User  Jury  saw  a  Simple  UIP  (SUEP),  and  the  team  sought  their 
feedback.  The  User  Jury  evaluated  the  SUIP  and  provided  input  regarding  the  alternatives  they 
were  shown.  The  SUIP  was  revised  until  the  jury  was  comfortable  with  it.  The  project  team 
developed  a  User  Interface  Style  Guide  (UISG)  to  codify  the  UIP’s  requirements,  and  all  further 
prototype  development  was  consistent  with  the  UIP. 

The  project  team  continued  to  collect  requirements  data  from  Task  3,  and,  with  interface 
requirements  determined,  was  ready  to  begin  development  of  operational  versions  of  the  CITT 
prototype.  (Although  the  Research  Program  Plan  had  specified  that  prototype  development 
would  begin  when  design  was  80%  complete,  development  actually  began  much  earlier.)  The 
team  began  to  identify  critical  functions  and  capabilities  and  to  define  the  sequence  of  tasks  that 
end-users  were  expected  to  complete  using  the  CITT.  At  this  point,  the  team  specified 
preliminary  database  requirements  and  preliminary  database  designs. 

Next,  the  team  developed  a  software  architecture  description  in  which  the  CITT  was 
partitioned  into  major  subsystems,  interactions  among  subsystems  were  specified,  and  plans  to 
produce  them  were  constructed.  At  this  point  prototype  versions  began  to  emerge,  and  the  team 
developed  architecture  considerations  for  both,  deciding  that  the  actual  software  for  both 
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versions  would  be  developed  in  staged  builds.  Builds  are  small  incremental  product 
development  steps  aimed  at  producing  running  code  as  soon  as  possible  (Zimmerman,  1997). 
More  precisely,  McConnell  (1998)  identifies  a  build  as  a  specific  instance  of  a  software  program 
at  a  particular  time  during  its  development.  Since  actual  development  depends  on  stable 
requirements,  breaking  a  software  project  into  builds  allows  developers  to  code  one  build  and 
incorporate  changes  into  the  next  or  a  later  build.  Each  build  supports  succeeding  builds  and/or 
improves  requirement  knowledge.  Within  the  staged  builds  of  the  various  CITT  software 
components,  the  project  team  constructed  several  incremental  builds  to  arrive  at  the  CITTSA  and 
CITTDT  versions  that  were  eventually  tested. 

The  first  step  in  each  staged  build  was  documenting  the  objectives  and  goals  for  that 
particular  build  and  identification  of  which  new  functions  or  capabilities  would  be  included  in 
the  build.  While  planning  subsequent  builds,  the  team  considered  significant  problems  (bugs)  or 
defects  identified  in  previous  builds,  however,  they  found  that  it  was  not  always  possible  to 
address  all  existing  problems  in  the  succeeding  build,  so  some  issues  were  deferred  until  the  final 
build.  With  the  requirements  for  a  build  identified,  the  next  task  was  to  design,  code,  and  test  the 
build.  Wherever  possible,  software  was  developed  and  tested  in  modules  (smaller  subsets  of 
functional  code),  then  integrated  and  carefully  tested.  Specific  details  on  the  development  of 
each  prototype  version  are  covered  next. 

CITT  Standalone 


Database  application  development 

The  life  cycle  development  for  the  CITTSA  database  application  consisted  of  four 
structured  builds  with  construction  in  Microsoft®10  Access  97  and  mainly  addressed  the  Produce 
Training  Materials  activity  of  the  CITT  model.  Build  0  focused  on  design  and  paper-based  drafts 
of  tables,  relationships,  and  application  forms  needed  before  beginning  physical  construction  of 
the  application.  It  also  included  partitioning  application  activities  over  the  subsequent  three 
builds.  Build  1  focused  on  construction  and  population  of  data  tables  and  fields,  development  of 
forms  for  the  Select  an  Exercise  and  View  an  Exercise  activities,  and  design  and  construction  of 
forms  for  the  Modify  an  Exercise  activity.  Build  2  marked  the  first  effort  of  error  correction  and 
continued  the  application  development  with  implementation  of  code  enhancements  in  the  Select, 
Review  and  Modify  modules,  construction  of  the  Create  an  Exercise  module,  development  of 
reports,  population  of  all  data  required  in  CITTSA,  and  integration  of  utility  features.  Build  3 
was  primarily  intended  to  be  an  error  correction  build  but  was  also  used  for  the  addition  of  utility 
features  and  developing  forms  for  the  instruction-oriented  activities  outlined  in  the  CITT 
dynamic  data  exchange  (DDE)  document.  Development  progressed  at  a  reasonable  pace 
following  the  delivery  strategy  as  intended,  and  Builds  1-3  concluded  with  a  CITTSA  version 
released  for  evaluation. 

Build  0.  The  planning  activities  that  comprised  Build  0  were  critical  to  ensuring 
application  functionality  and  were  organized  to  coincide  with  the  progression  of  activity 
modeling.  During  the  activity  modeling  process,  STRUCCTT-2  exercises  were  analyzed,  broken 
into  data  entities  (tables)  and  attributes  (fields),  and  a  draft  database  table  and  relationship 


10  Microsoft  is  a  registered  trademark  of  Microsoft  Corporation. 
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structure  were  designed.  Using  analyses  performed  during  activity  modeling  and  application  of 
first-through-fourth  normal  form  rules,  the  team  refined  the  draft  structure  of  the  CUT  database. 
(For  a  discussion  of  normal  form  rules  as  used  to  test  the  structure  and  integrity  of  data  models, 
see  Reingruber  and  Gregory,  1994.)  Initial  sketches  of  user  interface  forms  that  included  fields 
and  controls  were  drawn  for  each  activity  in  the  Select  an  Exercise,  View  an  Exercise,  and 
Modify  an  Exercise  activities.  First  draft  forms  were  built  in  a  Build  0  database  for  the  main 
menu  which  addressed  the  Use  the  Commanders’  Integrated  Training  Tool  activity,  and  for  each 
of  the  modules  directly  below  that  activity:  Navigate  CITT,  Learn  About  CCTT,  Produce 
Training  Materials,  Produce  Exercise  Files,  and  Execute  Support  Functions.  Build  0  ended  with 
a  lock  placed  on  the  activities  that  the  CITTSA  would  perform  with  the  exception  of  Create  an 
Exercise  that  would  be  modeled  at  the  end  of  Build  1 . 

Build  1.  Build  1  tested  the  materials  drafted  in  Build  0  and  initiated  application 
development  by  addressing  three  activities:  Select,  Review  and  Modify  which  are  included  in  the 
Produce  Training  Materials  module  (See  Figure  14).  The  data  storage  tables11  for  the  application 
were  built  in  a  Build  1  database,  and  relationships  were  created  between  tables  to  handle  data 
integrity.  The  team  applied  data  from  two  exercise  TSPs  into  the  tables  to  verify  data  storage 
capabilities.  Next,  they  evaluated  data  retrieval  and  update  capabilities  by  developing  the  Event 
Guide  and  Workstation  Operator  Guidelines  forms  in  the  Modify  an  Exercise  activity.  During 
this  effort,  an  Access  97  limitation  disallowing  more  than  two  levels  of  embedded  sub-forms 
forced  the  redesign  of  components  on  both  forms. 


CITT  Overview 


£  HowTo 
Navigate  CITT 


Leani  About 
I  CCTT 


£■ .  Produced- 
fvrTrainirig^^: 

f:  Materials 


f  Produce., 
-Exercise  Files 

4  • 


Exercise 
Management 
Tools  ; 


ExHCfTT 


Commanders*  Integrated  Training  Tool  for  the 
Close  Combat  Tactical  Trainer  ; 

aTT&udoTo>Jt  | 


Welcome  to  CTTT.  Jt 

CfTT  is  a  fully  integrated  training  and  training  management  system  that  supports  aJI  features  o<  unit  framing  „ 
in  the  Close  Combat  Tactical  Trainer.  It  indudes  exerrise  selection  end  development  extensive  * 

supporting  information  and  help,  navigation  aids,  and  exerrise  management  functions  for  a  veriefy  o*  users.  »■ 

For  Unit  Commanders/Leaders.  it  provides  a  tool  for  selecting  exercises  fortraining  in  CCTT  that  match  " 
the  units  current  training  needs.  H  no  existing  exerrise  adequately  supports  the  unifs  troinmg  needs.  CfTT  * 
also  provides  tools  for  modifying  existing  exercises  or  for  creating  new  exercises  that  do  satisfy  ft®  needs. 

-  For  CCTT  exercise  Observer/Controllers,  it  provides  information  on  all  O/C  functions  involved  in  CCTT  " 
training  as  well  as  information  on  conducting  the  After  Action  Reviews  which  ore  critical  to  successhi 
structured  training.  In  addition.  CITT  provides  the  O/C  access  to  all  materials  necessotyto  direclfte  unit  ~ 
as  they  prepare  for  and  execute  training  in  the  CCTT. 

\~  For  other  workstation  operators  (e  g.  Combat  Engineering  Support  Field  Artillery  Battalion  Tecfcol  * 
Operation  Center,  etc.),  it  provides  information  on  workstation  operation  and  use  as  wef  as  the  spedfc 
information  needed  by  each  workstation  operator  to  support  the  unifs  training  on  specific  exerrises 
For  CCTT  Site  Staff,  it  provides  the  materials  required  to  build  exercises  and  manage  exerase 
aexecution.  !•' 

I-  For  anyone  interested  in  learning  oboutthe  Close  Combat  Tactical  Trainer.  Structured  Training 
IPrindples.  and  the  application  of  structured  training  principles  to  CCTT  exercise  development  it  provides 
lextensive  information  on  each  of  these  topics.  1 


Figure  14.  The  CITTSA  Main  Menu  screen. 


11  A  table  can  be  conceptualized  as  a  row  x  column  matrix  where  the  columns  represent  data 
elements  and  the  rows  represent  specific  records.  The  final  CITTSA  included  38  tables. 
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The  team  developed  the  Modify  Training  Objectives  form  with  wizard  functionality 
which  includes  a  graphic  map  or  tab  navigation  and  tab-specific  assistant  help  (See  Figure  15). 
The  remaining  components  of  modify  were  constructed  and  sequenced  with  previous  and  next 
buttons  to  open  and  close  forms  as  the  user  proceeds  through  the  module. 


Figure  15.  The  CITTSA  Modify  Training  Objectives  screen. 

The  project  team  designed  the  Select  an  Exercise  forms  to  allow  the  user  to  locate  an 
exercise  by  name,  by  echelon  and  unit,  by  mission,  or  by  up  to  five  ARTEP  -  MTP  tasks  specific 
to  the  user’s  echelon  and  unit.  After  building  these  forms  with  the  inclusion  of  selection  by 
author  code  or  selection  by  type  of  exercise  on  the  Select  by  Name  form,  the  development  then 
focused  on  the  forms  required  to  review  an  exercise.  This  process  involved  creating  a  form  for 
each  section  of  the  exercise  and  reusing  the  Event  Guide  and  Workstation  Operator  Guidelines 
forms  and  many  of  the  sub-forms  developed  for  modify  so  that  error  correction  and  design 
enhancements  would  involve  only  one  source  form.  The  entire  collection  of  review  forms  was 
appended  at  the  end  of  the  modify  module  with  full  editing  capabilities  so  that  the  user  could  edit 
an  exercise  based  on  the  designed  CITT  methodology  or  at  his  discretion. 

The  final  effort  in  Build  1  was  writing  the  code  necessary  to  copy  all  the  records 
associated  with  an  exercise.  This  was  a  particularly  complex  coding  effort  since  each  exercise 
spans  fifteen  tables,  fourteen  of  which  contain  multiple  data  records  for  each  record  in  a  parent 
table.  Build  1  was  completed  with  a  release  of  the  Build  1  application  for  testing  by  the  CITT 
team,  production  of  screen  shots  for  User  Jury  review,  and  completion  of  the  modeling  and 
design  activities  for  the  Create  an  Exercise  module. 

Build  2.  Development  efforts  in  Build  2  alternated  between  a)  production  of 
functionality  for  required  modules,  b)  addition  of  features  requested  by  team  members,  the  CITT 
User  Jury,  or  government  representatives,  and  c)  correction  of  errors  uncovered  during  testing  by 
the  project  team.  The  final  module  developed  in  CITT  was  the  Create  an  Exercise  component. 
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This  included  a  wizard  for  the  creation  of  a  set  of  exercises,  the  code  necessary  to  create  as  many 
copies  of  the  template  exercise  TSP  as  the  user  required,  design  of  new  forms  for  Create,  and 
reusing  of  the  modify  module  forms  to  ensure  consistency.  The  team  constructed  reports  to  print 
or  preview  all  exercise  materials,  developed  a  menu  bar  and  toolbar,  integrated  identifiers  for 
help  topics  into  all  forms,  and  created  the  CITT  agent  to  provide  audio  and  text  information  with 
animation.  With  the  user  interfaces  functioning,  the  team  populated  the  database  with  the 
remaining  STRUCCTT  exercise  TSPs,  and  filled  the  lookup  tables  with  data  for  Mission 
Training  Plan  (MTP)  tasks,  task  steps  and  SAF  workstation  combat  instruction  sets  (CISs).  After 
correcting  known  bugs,  the  team  released  the  Build  2  database  for  informal  user  testing. 

Build  3.  The  initial  timeline  for  CITTSA  application  development  reserved  Build  3  for 
integration,  testing  and  debugging.  Errors  found  during  testing  consisted  of  users  unable  to  add 
items  to  forms  based  on  queries;  records  remaining  in  the  tables  after  related  but  not  linked  data 
were  deleted;  and  a  variety  of  correctable  functionality  and  presentation  errors.  Errors  were 
recorded  in  the  Census  97® 12  defect  tracking  software  and  assigned  to  an  owner  who  fixed  the 
error,  annotated  a  summary  of  the  repair,  and  closed  the  recorded  defect.  Integration  efforts 
involved  repackaging  the  Map/Overlay  Tool  for  use  in  modifying  a  CITT  overlay  or  training 
event  diagram;  integrating  the  help  files  and  the  Instructional  Overview  of  CCTT  module;  and 
adding  functionality  to  the  Navigate  CITT,  Produce  Exercise  Files,  and  Execute  Support 
Functions  forms.  The  Build  3  database  was  re-tested  by  the  project  team  before  its  release  for 
formative  evaluation. 

The  release  of  CITT  in  staged  builds  allowed  the  development  team  the  opportunity  to 
find  and  fix  errors  and  provided  a  working  complement  to  the  CITT  design  models.  The 
selection  of  Microsoft  Access  97  as  the  primary  development  program  proved  to  be  an 
acceptable  choice  since  Access  97  has  the  flexibility  to  allow  structural  changes  without  losing 
data  and  provided  a  programming  environment  capable  of  meeting  the  functionality 
requirements.  A  SQL  Server-Visual  Basic®13  solution  should  be  considered  for  future 
development  owing  to  the  limitations  of  Access  97.  These  limitations  include  no  more  than  two 
levels  of  nested  subforms,  data  on  forms  not  responding  to  filtering  or  sorting  settings,  unreliable 
functioning  of  hyperlinks  stored  within  tables,  and  embedded  objects  occupying  excessive 
storage  space  when  a  record  is  copied.  Access  97  proved  to  be  an  effective  model  for  migrating 
to  SQL  Server  and  the  programs  used  in  developing  the  ClTl'DT.  Development  of  the  CITTSA 
guided  the  design  and  development  of  the  Ci'lTDT  and  illuminated  areas  of  the  activity  models 
needing  additional  definition. 

Help  File  Development 

A  key  consideration  in  the  design  and  development  of  the  CITT  was  that  it  must  assist  the 
users  in  accomplishing  the  tasks  that  they  want  to  perform.  The  CITT  SOW  specified  that  the 
design  include  user  guidance  in  the  form  of  “help”  screens  and  other  user-oriented  aids  to  be 
incorporated  directly  into  the  CITT. 


12  Census  97  is  a  registered  trademark  of  MetaQuest  Software,  Inc. 

13  Visual  Basic  is  a  registered  trademark  of  Microsoft  Corporation. 
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In  the  prototype,  the  team  accomplished  this  by  developing  and  including  a  comprehensive, 
integrated  help  system.  The  general  steps  involved  in  developing  a  help  system  include: 


1 .  Create  the  help  topics  and  structure. 

2.  Create  project  files. 

3.  Create  contents  and  index  files. 

4.  Compile  the  help  file. 

5.  Test  the  help  file. 

6.  Integrate  the  help  system  with  the  main  application. 

Create  the  Help  Topics  and  Structure.  One  of  the  first  steps  taken  in  the  development  of 
the  help  system  was  to  review  existing  software  programs  and  manuals  to  identify  different  ways 
in  which  on-screen  help  can  be  presented  to  the  user.  A  key  reference  during  this  step  was  The 
Windows  Interface  Guidelines  for  Software  Design  (Microsoft  Corp.,  1995b).  Based  on  this 
review,  the  project  team  determined  that  there  were  two  main  forms  of  help  that  were  most 
applicable  to  the  CITT:  content  help  and  task-oriented  help.  Content  help  is  best  described  as 
providing  users  with  background  information  that  assists  them  in  more  fully  understanding  the 
activity  or  process  they  are  performing  (see  Figure  16).  Task-oriented  help,  on  the  other  hand, 
provides  step-by-step  instructions  on  completing  an  activity  or  task  (see  Figure  17). 
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Create  a  New  Exercise  -  Define  Mission  Set  Parameters 

The  first  step  in  creating  a  new  exercise  is  to  define  the  mission  set  parameters.  These  parameters  are  the 
foundation  for  the  mission  set  and  help  guide  the  creation  of  the  exercises  within  the  set.  The  Define  Mission  Set 
Parameters  consists  of  the  following  components: 

♦  Establish  Initial  Setting 

♦  Select  Mission  Set  Tasks 

♦  Determine  Mission  Set  Concept 

♦  Generate  OPORD  Materials 

♦  Partition  into  Exercises 


What  is  a  Mission  Set? 

Mission  Sets  contain  exercises  that  provide  units  the  opportunity  to  execute  collective  tasks  within  the  context  of  a 
battalion  or  squadron  level  scenario.  The  battalion  or  squadron  level  scenario  is  partitioned  into  segments  that  make 
up  the  framework  of  the  platoon  ,  companyrteam,  or  troop  exercises.  Each  segment  can  be  developed  into  an 
exercise  that  wffl  have  events  that  occur  which  cue  the  tasks  which  are  the  focus  of  the  exercise. 


Figure  16.  Content  help. 
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With  the  methodology  for  providing  help  determined,  the  next  step  was  to  develop  a 
technique  for  ensuring  that  the  help  would  be  comprehensive.  The  team  considered  two  ways: 
develop  help  for  each  individual  screen  in  the  CITT,  or  develop  help  for  each  major  activity  the 
user  would  perform.  The  team  selected  the  second  method  based  on  the  assumption  that  activity 
based  help  would  cover  all  of  the  information  the  user  would  need  to  perform  the  various  tasks 
available  in  the  CITT.  To  ensure  that  the  help  would  attend  to  both  content  and  task-oriented 
information,  the  team  developed  a  worksheet  to  assist  the  team  in  identifying  the  type  of  help 
required  for  each  activity  (content,  task-oriented,  or  both)  and  to  ensure  that  all  activities  were 
covered  (see  Figure  18). 
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The  Commanders*  integrated  Training  tool  BSk  pjfiijl  E3 


Help  Topics  Back  I Options 


To  Change  the  Sequence  of  Existing  Events 


1 .  Place  the  cursor  in  the  appropriate  event  number  box  that 
you  wish  to  change. 

2.  Type  in  the  new  sequence  number  for  the  event. 

3.  Continue  to  renumber  the  sequence  of  events  as  required. 

Note 

Changing  the  sequence  of  existing  events  is  not 
recommended.  The  events  will  be  updated  to  the  event 
descriptions  in  the  exercise  Overview  and  Event  Guide. 
However,  simply  changing  the  sequence  of  events  will 
affect  a  significant  number  of  components  that  are  not 
automatically  updated.  A  better  course  action  would  be  to 
select  another  exercise  or  create  an  exercise  that  better 
meets  your  needs. 


Related  Topics 

To  Change  the  Sequence  of  Existing  Events 


Figure  17.  Task-oriented  Help. 

The  first  component  of  the  CITT  for  which  help  was  developed  was  Select/Review  an 
Exercise.  This  module  was  less  complex  than  the  Modify  an  Exercise  and  Create  an  Exercise 
modules.  The  team  then  examined  the  Select/Review  an  Exercise  module  of  the  CITT  prototype 
activity  by  activity,  making  very  detailed  notes  about  the  procedure  involved  in  completing  an 
activity.  Next,  they  recorded  this  information  in  the  Task  Help  column  of  the  worksheet,  then 
finished  the  Content  Help  column  by  identifying  the  information  the  user  would  need  to  assist 
him  or  her  in  completing  the  activity.  The  worksheet  proved  to  be  very  helpful  in  writing  the 
help  for  Select/Review  an  Exercise.  It  assisted  in  clearly  and  thoroughly  identifying  what 
information  the  user  would  need  to  be  able  to  successfully  use  CITT. 
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Once  completed,  the  worksheet  served  to  guide  development  of  the  help  content  as  the 
team  wrote  step-by-step  instructions.  The  worksheet  was  used  extensively  for  Select  an 
Exercise,  however,  for  subsequent  modules,  the  process  became  more  of  a  mental  one,  and  the 
worksheet  was  not  used.  After  completing  the  Help  for  Select/Review  an  Exercise,  the  team 
wrote  help  for  the  following  two  components:  Modify  an  Exercise  and  Create  an  Exercise.  After 
writing  help  for  these  major  components,  they  reviewed  it  for  content  and  accuracy. 
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Figure  18.  Help  Worksheet. 

An  integral  part  of  the  development  of  the  help  material  was  the  on-going  dialogue 
among  the  project  team  particularly  between  the  software  developers  and  the  author  of  the  help 
material.  Frequent  discussions  with  the  software  developers  enabled  the  author  to  understand 
exactly  how  a  certain  function  of  CITT  was  to  work  ensuring  that  the  information  in  the  help 
material  was  accurate. 

Create  Project  Files.  The  help  materials  were  written  in  Microsoft  Word  format  and  were 
then  converted  and  imported  into  RoboHELP®14’  a  full-featured  help-authoring  package  for 
developing  Windows®  15-based  help  systems  (Blue  Sky  Software  Corporation,  1997a). 
RoboHELP  works  in  conjunction  with  Microsoft  Help  Workshop  to  simplify  the  development 
process  by  maintaining  help  content  and  project  files  and  by  integrating  those  files  to  develop  the 
help  file. 

Once  the  content  files  were  imported,  source  files  and  project  files  needed  to  develop  the 
help  file  were  created.  Source  files  contain  the  text  and  graphics  that  appear  in  the  help  topics 
(Topic  Files),  information  about  the  location  of  topic  and  graphic  files,  specifications  on  how  the 
help  windows  should  look,  and  settings  that  customize  the  way  the  help  file  functions  (Project 
Files).  The  files  used  to  develop  the  help  system  were  designed,  configured,  and  developed  in 


14  RoboHELP  is  a  registered  trademark  of  Blue  Sky  Corporation. 

15  Windows  is  a  registered  trademark  of  Microsoft  Corporation. 
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accordance  with  the  Windows  guidelines  (Microsoft  Corp.,  1995b)  and  are  consistent  with  the 
specifications  outlined  in  the  Programmer’s  Manual  for  the  CITT  Prototype  (Standalone 
Version)  (CITT  Team,  1998c).  The  result  of  using  these  guidelines  was  the  development  of  a 
help  system  that  was  consistent  with  current  standards. 

Create  Contents  and  Index  Files.  Two  important  features  of  Windows-based  help 
systems  are  the  Contents  and  Index  tabs.  The  Contents  tab  provides  users  with  a  hierarchical 
view  of  the  help  system  and  acts  as  a  table  of  contents  for  the  help  file.  Furthermore,  it  provides 
the  user  with  a  simple  mechanism  for  browsing  related  topics.  Windows-based  help  systems  use 
a  books/pages  metaphor  to  represent  categories  and  topics  in  the  Contents  tab  (see  Figure  19). 
Book  icons  represent  a  category  or  group  of  related  topics,  and  a  page  icon  represents  an 
individual  topic.  The  table  of  contents  was  developed  based  on  the  major  CITT  functions  and 
the  tasks  that  comprise  them. 


Click  a  book,  and  then  click  Open.  0(  click  another  tab,  such  as  Index. 


Welcome  to  CITT 


(jQ)  CITT  Users'  Manual 

*  Learn  About  CITT 
How  to  Navigate  CITT 

^  Learn  About  CCTT 
(Q  Produce  Training  Materials 
[?]  Produce  T raining  M aterials 
[?)  CITT  Training  Materials 
^  View  an  Exercise 
^  Select  an  Exercise 
Modify  an  Exericse 
Create  an  Exercise 

*  Produce  Exercise  Files 

^  Exercise  Management  Tools 
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Figure  19.  Help  Contents  Screen. 

An  Index  tab  was  created  to  provide  users  with  a  list  of  all  of  the  topics  available  in  the 
help  system  organized  alphabetically  by  title  and  keyword  (see  Figure  20).  The  index  provides  a 
means  to  browse  quickly  through  the  topics  to  locate  desired  information. 
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Compile  the  Help  File.  After  the  topic  files,  project  files,  contents,  and  index  files  were 
created,  they  were  compiled  using  Microsoft  Help  Workshop.  Applying  a  technique  called  data 
compression,  the  Help  Workshop  program  packs  topic  files,  graphics,  and  other  project  files  into 
a  single  file  that  users  can  display.  This  process  ensures  that  the  compiled  help  file  is  as  small  as 

possible. 

During  the  compression  process,  a  full-text  search  capability  was  developed  and  a  Find 
tab  was  created.  This  allows  users  to  search  for  any  word  or  phase  contained  in  the  help  file  (see 
Figure  21).  At  the  same  time  the  help  compiler  examines  topic  files,  character  formatting,  and 
hyperlinks  and  translates  them  into  viewable  formats. 


Help  f  opics:  The  Coinmandets'  lntetgtated  Training  Tool 
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Figure  20.  Help  index  screen. 

During  the  compilation  process,  the  help  compiler  performed  an  internal  test  of  the 
project  files.  When  it  located  programmatic  errors  (coding  errors),  it  printed  a  waming/error 
message  and  continued  with  the  compilation.  With  these  errors  corrected,  the  project  files  were 
recompiled  until  all  programmatic  errors  were  discovered  and  corrected. 

Test  the  Help  File.  After  correcting  all  programmatic  errors  and  recompiling  the  help 
file,  the  team  tested  the  help  system  for  logical  errors.  Logical  errors  are  programmatically 
correct,  however  they  do  not  perform  as  intended  (e.g.,  they  link  to  the  wrong  document).  These 
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errors  were  discovered  though  a  systematic  testing  process  that  addressed  six  areas  of  help 
testing  outlined  in  the  Microsoft  Windows  95  Help  Authoring  Kit  (Microsoft  Corp.,  1995a): 

1 .  Cleaning  up  all  compiler  messages. 

2.  Checking  the  index,  spelling,  and  titles. 

3.  Checking  formatting  and  styles. 

4.  Reviewing  accuracy  and  style  of  content. 

5.  Accessing  help  from  the  interface. 

6.  Testing  jumps,  pop-ups,  and  browse  sequences. 


Help  Topics:  The  Commanders*  Integrated  Training  Tool 
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Figure  21.  Help  find  screen. 

This  testing  process  uncovered  a  number  of  additional  errors  that  were  corrected.  Internal  testing 
of  the  help  file  was  a  time-consuming  and  tedious  process,  however,  it  was  one  of  the  most 
important  steps  in  insuring  that  the  help  file  operated  as  intended. 

Integrate  the  Help  System  with  the  Main  Application.  During  the  development  process,  a 
unique  ID  was  assigned  to  each  topic  that  made  it  possible  for  the  main  (database)  application  to 
find  and  access  the  topic.  These  topic  IDs  were  programmed  into  the  main  application.  This 
process  connected  the  help  file  to  the  CITT  application  program  and  allowed  access  to  the  help 
file.  Once  the  topics  were  linked  to  the  application,  a  systematic  testing  process  was 
implemented  and  each  screen  of  the  application  was  accessed  by  the  project  team,  and  its 
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associated  help  topic  was  displayed.  This  process  validated  the  integration  process  and  revealed 
any  missing  links.  Once  an  error  was  discovered,  it  was  recorded  in  the  Census  97  defect 
tracking  software  and  fixed  in  the  help  file.  The  project  team  applied  this  testing  process  to  the 
complete  help  system,  and  they  conducted  additional  testing  of  the  integrated  help  system  during 
CITT  informal  user  testing  and  formative  evaluation. 

Integrate  the  Instructional  Overview  into  the  CITTSA 

The  final  major  activity  in  the  development  of  the  CITTSA  was  the  incorporation  of  the 
Instructional  Overview  content  into  the  “Learn  About  CCTT”  module  of  the  CITT.  The 
development  and  production  of  the  content  material  was  previously  described  under  Task  2. 

To  incorporate  the  material  into  the  CITT,  the  Word  files  that  were  produced  in  Task  2 
were  converted  to  HyperText  Markup  Language  (HTML)  files  using  Microsoft  FrontPage  98®16. 
During  the  conversion  process,  errors  were  produced  in  converting  Word  formats  to  HTML. 
FrontPage  98  and  Microsoft  Visual  Interdev  17  were  used  to  correct  these  errors  as  well  as  to 
insert  additional  HTML  code,  graphics,  and  scripts.  Microsoft  Visual  Basic®  5.0  and  Microsoft 
Script  Debugger  1.01  were  used  to  test  the  source  scripts  before  they  were  inserted  into  the 
HTML  code. 

Next,  the  HTML  files  were  imported  into  RoboHTML®18,  which  is  a  development  tool 
based  on  Microsoft’s  HTML  Help  technology  (Blue  Sky  Corporation,  1997b).  Microsoft’s 
HTML  Help  is  a  set  of  standards  for  Help  systems  based  on  the  HTML  format  and  is  intended  to 
provide  an  alternate  way  to  display  help  materials  (Wexler,  1998).  An  important  advantage  of 
the  HTML  Help  technology  is  its  ability  to  take  advantage  of  the  latest  Internet  technologies 
such  as  Active-X  controls,  Scripts,  and  multimedia  effects. 

Having  imported  files  into  RoboHTML,  the  team  used  them  to  develop  the  Table  of 
Contents  that  outlines  the  general  structure  of  the  Instructional  Overview  and  an  index  providing 
keyword  search  functionality.  They  then  compiled  the  files  to  create  the  working  version  of  the 
10  which  provides  the  same  functionality  as  described  previously  for  the  help  system:  full  text 
search  capabilities,  an  index  of  keywords,  and  a  table  of  contents.  Figure  22  displays  a  typical 
screen  for  the  10. 


CITT  Distributed 


Database  Application  Development 

Although  original  planning  specified  staged  builds  for  the  CITI  DT  (as  was  described 
above  for  the  CITTSA),  this  did  not  occur.  Development  of  the  CITTDT  did  not  begin  until  well 
after  development  of  the  CITTSA,  and  once  it  did  begin,  there  was  insufficient  time  to  employ  a 
staged  build  process.  The  delay  occurred  for  several  reasons: 


16  FrontPage  98  is  a  registered  trademark  of  Microsoft  Corporation. 

17  Visual  Interdev  is  a  trademark  of  Microsoft  Corporation. 

18  RoboHTML  is  a  registered  trademark  of  Blue  Sky  Software,  Inc. 
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1 .  A  deliberate  decision  was  made  to  focus  on  development  of  the  CITTSA  since  the  project 
team  thought  there  was  greater  likelihood  that  it  could  be  completed  successfully. 

2.  The  work  that  the  primary  application  programmer  for  the  CITTSA  was  completing  as 
described  above  for  Build  0  was  also  applicable  to  the  CITTDT  (i.e.,  Identification  of  data 
entities  and  attributes,  the  database  table  and  relationship  structure,  user  interface  forms), 
thus  it  was  inefficient  to  start  on  the  CITTDT  until  Build  0  was  complete.  Since  Build  0 
included  some  application  development,  this  gave  development  of  the  CITTSA  a  head  start 
over  the  development  of  the  CITTDT. 

3.  The  CITTSA  application  programmer  was  an  active  participant  in  the  CITT  design  process 
as  opposed  to  the  CITTDT  developers  who  participated  in  only  a  few  of  the  design  meetings 
Familiarity  with  the  design  facilitated  development  of  the  CITTSA  and  led  to  a  decision  to 
develop  the  CITTDT  based  on  the  CITTSA. 
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Figure  22.  The  CITT  Instructional  Overview. 

The  development  of  the  CITTDT  began  once  the  CITTSA  design  and  database  structure 
were  approximately  90%  complete.  An  initial  decision  in  the  process  concerned  the 
development  software.  Access  was  ruled  out  since  it  does  not  adequately  support  a  multi-user 
environment  that  would  be  required  in  a  distributed  application.  Instead,  Microsoft  SQL  was 
chosen  which  necessitated  converting  the  CITTSA  database  to  SQL.  This  proved  to  be  a 
complex  process  which  involved  upsizing  the  Access  database  into  SQL  using  Microsoft  Access 
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Upsizing  Tool.  With  upsizing,  a  report  was  generated  that  gave  the  programmers  valuable 
information  on  required  changes  to  the  table  structures.  This  was  particularly  helpful  since  it 
allowed  table  design  modification  to  be  kept  to  a  minimum. 

PowerPoint®19  files  (e.g.,  maps,  overlays)  and  Word  files  (e.g.,  Operations  Orders) 
included  in  the  CITTSA  were  not  imported  into  SQL  6.5  due  to  the  inability  of  the  database  to 
store  binary  large  objects  .  When  the  Microsoft  SQL  7  Beta  became  available,  this  limitation 
disappeared.  The  CITTSA  data  was  again  imported  but  this  time  into  SQL  7,  and  the  MSWord 
and  PowerPoint  files  were  converted  into  binary  large  objects. 

A  computer  system  (server)  allowing  development  and  access  to  the  CITTDT  was  placed 
in  Louisville,  KY  for  full  time  Internet  access  while  the  application  was  being  developed.  This 
allowed  the  CITTDT  application  developers  to  easily  access  the  system  for  development, 
reduced  fears  that  a  server  crash  would  have  adverse  impact  on  other  servers,  and  allowed 
members  of  the  project  team  at  Fort  Knox  to  determine  how  well  the  CITTDT  would  perform  on 
the  actual  Internet.  As  will  be  discussed  below,  the  project  team  determined  that  Internet  access 
speed  varies  widely  depending  on  time  of  day  and  quality  of  the  Internet  connection. 

The  Review  and  Create  components  of  the  CITTDT  were  developed  using  Microsoft 
Active  Server  Pages  in  conjunction  with  queries  from  the  SQL  7  database.  The  Active  Server 
Pages  utilized  SQL  7  views  and  stored  procedures  and  ActiveX  control  technology.  The  Modify 
component  was  not  completed  for  the  CITTDT.  Figure  23  illustrates  the  CITTDT  Main  Menu 
screen. 

Help  File  Development 

The  development  of  the  help  files  for  the  distributed  CITT  was  an  extension  of  the 
development  process  for  the  standalone  system.  Once  all  files  were  fully  developed  and  tested 
for  the  standalone  help  system,  the  team  converted  files  into  a  format  suitable  for  a  web 
application. 

First,  the  help  topics  associated  with  the  CITT  application  were  converted  to  HTML. 

This  conversion  process  allowed  the  development  team  to  reuse  all  the  materials  previously 
developed  for  the  standalone  system  and  accounted  for  over  95%  of  the  files  needed  to  develop 
the  help  system  for  the  CITTDT. 

After  the  topic  files  were  converted,  they  were  integrated  into  a  navigation  system.  The 
navigation  system  is  similar  in  functionality  to  the  tree- view  control  used  in  the  standalone 
Instructional  Overview.  In  order  to  provide  the  same  functionality  on  the  two  systems,  the 
distributed  CITT  utilizes  Java  applet  technology.  Therefore,  a  server-side  HTML  Help  Java 
Applet  was  programmed  to  work  with  the  help  system.  Figure  24  demonstrates  the  results  of  the 
Java  applet. 


19  PowerPoint  is  a  registered  trademark  of  Microsoft  Corporation. 


47 


elcome  to  Cut.  ~- 

CnT  is  a  fully  integrated  training  and  training  — ■ 

management  system  that  supports  all  features  of  unit 
training  in  the  Close  Combat  Tactical  Trainer.  It  includes 
exercise  selection  and  development,  extensive 
supporting  information  and  help,  navigation  aids,  and  > 
exercise  management  functions  for  a  variety  of  users.  % 

£ 

—  For  Unit  Commanders/Leaders,  it  provides  a  tool  for  \ 
selecting  exercises  for  training  in  CCTT  that  match  the  ^ , 
unit's  current  training  needs.  If  no  existing  exercise 
adequately  supports  the  unit’s  training  needs,  CTIT  also  $ 


•  MSAoertf  i  .5  Bee 


CGRAM  1.5  Exe 


MeriinSFX 


ToolTe 


Best  viewed  at  800  by  600  and  in  32k  colors  using  Microsoft 
TEA.  Click  the  hyperlink  to  Test  Your  Browser 


Tr1 


Figure  23.  The  CITTDT  Main  Menu  screen. 
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Figure  24.  HTMLHelp  Java  Applet  TOC. 


A  Help  Button  was  provided  on  each  screen  of  the  distributed  application  and  each  topic 
file  was  linked  with  its  associated  screen.  The  system  was  designed  to  allow  the  user  access  to 
the  table  of  contents  and  index  information  after  a  help  topic  is  accessed. 

Integrate  the  Instructional  Overview  into  the  CITTDT 

The  distributed  10  was  created  from  materials  previously  developed  for  the  standalone  10 
system.  Only  minor  modifications  were  required  because  the  standalone  10  was  developed  using 
web-based  technologies  (HTML,  Active-X,  VB  Script,  and  Java  Script).  These  modifications 
included  changing  the  navigation  control  and  removing  all  large  active-movie  video  clips 
because  of  excessive  download  times.  The  procedures  for  implementing  the  navigation  control 
were  identical  to  those  for  the  Java  applet  outlined  in  the  distributed  Help  system  section. 

After  the  files  were  completed,  they  were  uploaded  to  the  distributed  server  and  a  link  to 
the  10  was  added  to  the  main  application.  Since  the  distributed  10  was  developed  as  a  self- 
contained  product,  the  integration  only  required  placing  a  button  to  access  the  main  page.  After 
the  10  was  uploaded,  the  development  team  tested  the  functionality  of  the  system  and  corrected 
errors. 


During  CITT  prototype  development,  ten  user-initiated  installation  programs  to  support 
CITT  prototype  research  development  efforts  were  developed.  CITTSA  installs  were  delivered 
by  CD-ROM  and  CITTDT  installs  were  delivered  by  download.  User-initiated  installation 
programs  were  developed  using  InstallShield  Professional®20  5.1  and  InstallShield  Package  For 
The  Web®21  2.02. 

Figure  25  illustrates  the  final  CITT  prototype  as  implemented.  The  figure  applies  to  both 
the  CITTSA  and  CITTDT  versions,  however,  not  all  modules  and  activities  were  implemented  in 
the  CITTDT  version.  In  addition,  a  representative  sample  of  annotated  screen  shots  from  the 
CITTSA  Modify  Exercise  and  Create  Exercise  activities  are  included  in  Appendix  C.  These 
screen  shots  provide  a  “feel”  for  the  CITT  system  as  it  is  would  be  used  to  produce  a  TSP. 


Task  5:  Develop  an  Implementation  Strategy  and  Fielding  Plan 

To  maximize  the  CITT's  effect  on  training  in  the  CCTT  across  the  Army,  training 
developers  must  create  an  effective  implementation  strategy.  This  strategy  must  take  into 
consideration  the  full  range  of  individuals  who  will  use  the  CITT  including  remote  users  and 
system  administrators.  In  addition,  the  strategy  must  address  compatibility  of  the  CITT  with 
Army  training  information  management  systems  and  databases  such  as  the  SATS  and  ASAT. 
This  section  will  examine  three  possible  methods  for  implementing  the  CITT: 

1.  CITT  fielded  specifically  to  support  CCTT. 

2.  CITT  fielded  as  a  standalone  component  of  SATS. 

3.  CITT  fielded  as  a  component  of  an  integrated  training  tool. 


20  InstallShield  Professional  is  a  registered  trademark  of  InstallShield  Corporation. 

21  InstallShield  Package  for  the  Web  is  a  registered  trademark  of  InstallShield  Corporation. 


49 


Development  of  a  strategy  that  incorporates  Cl  FT  into  the  Army  s  existing  automated  training 
management  architecture  will  offer  the  most  long-term  benefit  for  the  Army. 
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Figure  25.  The  CITT  prototype. 

The  CITT  Fielded  Specifically  to  Support  CCTT 

Since  the  CITT  is  designed  to  support  training  in  the  CCTT,  a  near-term  method  for 
implementation  would  be  fielding  it  specifically  to  support  CCTT.  This  would  entail  providing 
access  to  CITT  for  those  installations  that  have  CCTT  and  those  units  who  use  CCTT.  The  CITT 
would  be  placed  at  each  CCTT  site  including  mobile  sites  allowing  users  to  develop  training 
products  based  on  subject  matter  expertise  provided  by  the  CCTT  sites.  The  CITT  would  also  be 
placed  in  units  to  further  facilitate  the  unit’s  development  of  training  products  for  CCTT.  A 
cost-benefit  analysis  would  be  performed  to  determine  the  appropriate  echelon  at  which  to  place 
the  CITT.  This  alternative  is  illustrated  in  Figure  26. 
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Figure  26.  The  CITT  Fielded  to  Support  CCTT. 

To  support  sharing  of  training  materials  and  lessons  learned  for  training  in  CCTT 
throughout  the  Army,  an  army-wide  database  would  be  established  to  store  developed  training 
products  and  lessons  learned.  An  administering  agency  would  be  identified  to  manage  and 
maintain  the  database  allowing  units  to  review  training  materials  developed  by  other  units  as 
well  as  lessons  learned  from  training  in  CCTT.  To  support  units  in  obtaining  information,  a 
dedicated  replication  server  would  be  used  at  each  installation.  The  replication  server  would  also 
be  used  to  store  common  data  that  all  users  need  such  as  Learn  About  CCTT  or  EDUCCATT. 
The  administering  agency  would  receive  updates  from  ATSC,  PM-CATT,  and  TSM-CATT  to 
ensure  the  database  stays  current.  Updates  would  include  such  information  as  new  CCTT 
capabilities  or  updated  ARTEP-MTP  tasks. 

This  alternative  would  provide  units  with  access  to  the  CITT  allowing  them  to  produce 
training  materials  for  use  in  CCTT.  It  could  also  allow  CCTT  sites  to  assist  units  in  their 
preparation  of  training  materials.  This  alternative  could  be  implemented  relatively  quickly  with 
hardware  being  the  primary  expense.  However,  the  alternative  also  allows  for  little  or  no 
integration  with  existing  Army  training  information  management  programs.  It  would  create  an 
additional  training  program  that  users  would  need  to  leam  and  could  conflict  with  the  movement 
to  consolidate  training  management  programs. 

CITT  Fielded  as  a  Standalone  Component  of  SATS 

The  second  fielding  alternative  is  to  integrate  CITT  as  a  standalone  component  of  SATS. 
SATS,  based  on  the  Combined  Arms  Training  Strategy,  is  an  integrated  tool  that  enables  a 
trainer  to  develop  unit-specific  situational  training  and  manage  training  resources.  SATS 
provides  users  with  information  on  available  resources  and  helps  identify  training  priorities. 

Once  a  resource  has  been  identified  for  use,  SATS  includes  a  TSP  module  that  allows  users  to 
develop  a  TSP  for  a  training  event.  Figure  27  illustrates  this  alternative. 
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Integration  of  the  CITT  as  a  standalone  component  of  SATS  would  be  relatively  simple. 
As  users  indicate  their  intention  to  use  CCTT  and  enter  the  TSP  module,  SATS  would  link  them 
to  the  CITT.  Once  they  are  in  CITT  they  would  be  able  to  develop  a  TSP  to  support  their 
training  in  CCTT.  Under  this  alternative,  the  CITT  would  be  distributed  as  an  additional  CD- 
ROM  with  the  SATS  CDs. 
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Figure  27.  The  CITT  Fielded  as  a  Standalone  Component  of  SATS. 

Investigation  will  be  required  to  determine  the  best  method  for  linking  the  CITT  to  SATS 
including  the  information  that  could  be  carried  forward  from  SATS  into  CITT.  For  example, 
collective  tasks  identified  during  the  planning  of  training  could  be  integrated  into  the  CITT  TSP 
to  preclude  the  user  from  having  to  identify  collective  tasks  again  when  they  link  to  the  CITT. 

This  alternative  requires  compatible  hardware  and  software  for  CITT  and  SATS  and 
allows  for  the  fielding  of  CITT  as  a  component  of  SATS,  thus  avoiding  placing  an  additional 
training  information  management  program  into  the  Army's  training  architecture.  It  also  allows 
for  CITT  to  be  fielded  in  accordance  with  the  distribution  plans  for  SATS.  One  drawback  to  this 
method,  however,  is  that  it  would  provide  trainers  with  a  tool  that  addresses  only  one  training 
resource — CCTT. 
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Integrated  Training  Tool 


The  third  alternative  focuses  on  development  of  a  truly  integrated  training  tool  that 
allows  users  to  develop  training  products  for  different  training  events  occurring  in  live,  virtual,  or 
constructive  environments.  This  integrated  training  tool  would  also  include  information  that  is 
common  to  all  training  events  and  environments  and  would  support  a  variety  of  TSP  forma  s 
tailored  to  the  methods  of  training  in  different  environments.  Examples  of  common  information 
are  training  management  and  training  approaches.  It  would  be  possible  to  integrate  this  tool  into 
SATS  and  ASAT  providing  a  common  training  development  tool  for  the  Army  as  a  whole. 

Figure  28  illustrates  this  alternative. 
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Figure  28.  An  Integrated  Training  Tool. 

This  alternative  offers  the  advantage  of  building  on  existing  training  information 
management  programs  with  which  users  are  familiar  while  adding  additional  functionalities  that 
allow  the  user  to  develop  TSPs  to  support  a  variety  of  training  events. 

To  implement  this  alternative  an  administering  agency  would  be  identified.  A  logical 
candidate  would  be  the  ATSC  since  they  currently  administer  SATS  and  ASAT.  Since  the 
integrated  training  tool  could  be  a  component  of  SATS  and  ASAT,  it  would  be  fielded  based  on 
their  distribution  plans.  This  provides  access  to  the  integrated  training  tool  at  the  company  level 
for  units  in  the  field.  In  the  proponent  schools,  access  will  be  provided  to  the  appropriate 
agencies  tasked  with  training  development.  To  provide  access  to  developed  training  materials 
and  lessons  learned  throughout  the  Army,  products  developed  with  the  integrated  training  tool 
would  be  stored  in  an  Army- wide  database  such  as  the  Center  for  Army  Lessons  Learned  or 
ADTDL. 
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Figure  29  illustrates  a  possible  fielding  structure  for  this  alternative.  As  shown,  SATS  is 
fielded  to  the  company  level  and  higher.  Users  would  have  a  client  workstation  capable  of 
running  SATS  with  the  integrated  training  tool.  A  higher  headquarters  would  be  able  to  provide 
training  guidance  to  subordinates  and  review  training  materials  developed  based  on  that  training 
guidance.  Training  materials  or  exercises  that  are  determined  by  higher  headquarters  to  be 
effective  would  be  placed  in  the  Army-wide  database  after  being  approved  by  the  chain  of 
command. 
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Figure  29.  Integrated  Training  Tool  Fielding  Strategy. 

Users  at  proponent  agencies  would  use  the  same  integrated  training  tool  as  that  used  by 
units  in  the  field.  By  accessing  the  Army-wide  database,  users  at  the  proponent  would  review 
the  training  being  conducted  by  field  units  and  would  place  updated  training  information,  i.e. 
updated  collective  tasks  or  new  simulation  capabilities,  in  the  Army-wide  database.  This  would 
encourage  a  sharing  of  information  between  the  user  in  the  field  and  the  training  developer  at  the 
proponent. 

As  with  the  CITT  fielded  to  support  CCTT  alternative,  a  replication  server  would  be  used 
to  echelon  information.  By  storing  information  in  an  Army-wide  database,  the  efficiency  of 
user's  computers  will  be  improved.  Updated  training  information,  instructional  modules  on 
different  training  environments,  and  exercises  would  be  stored  in  the  database.  These  could  be 
downloaded  based  on  the  users’  needs  thus  allowing  them  to  tailor  the  information  on  their  local 
computer  to  best  support  their  needs. 

Users  would  have  the  application  loaded  on  their  computer  as  well  as  the  information 
they  have  downloaded  from  the  Army-wide  database.  This  provides  trainers  in  units  and  training 
developers  at  proponent  schools  an  identical  training  tool.  The  application  would  allow  them  to 
develop  training  materials  to  support  live,  virtual,  or  constructive  training  for  individual, 
collective,  or  leader  training  programs. 
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Common  Issues 


<  The  three  alternatives  discussed  above  share  several  fielding  issues.  One  involves  access. 
Under  each  alternative,  the  same  method  for  providing  access  to  remote  users  could  be  used. 
Users  would  register  on-line  with  the  administering  agency  and  receive  a  scaled  down  version  of 
the  application  on  CD-ROM.  Users  would  then  access  the  Army-wide  database  to  use  functions 
not  included  with  the  CD,  obtain  up  to  date  training  information,  and  view  exercises  in  the 
database.  As  users  develop  training  materials,  they  would  transfer  them  to  their  unit  for 
inclusion  in  the  Army-wide  database.  Transfer  of  training  materials  would  occur  via  electromc 
mail  or  disk  depending  upon  the  size  or  the  electronic  files.  This  is  illustrated  in  Figure  30. 


A  second  common  issue  concerns  connectivity  with  training  sites.  For  maximum 

effectiveness,  appropriate  training  sites,  such  as  CCTT  or  WarSim  2000,  would  be  equipped  with 
the  same  type  of  client  workstation  as  that  located  in  units  and  proponent  agencies.  This  would 
allow  units  to  transfer  developed  training  materials  to  the  training  site  that  will  support  them 
during  execution  of  the  training.  Ultimately  this  transfer  of  training  materials  would  result  in  a 
user  being  able  to  develop  a  virtual  or  constructive  exercise  and  send  it  directly  to  the  training 
site.  To  identify  ways  to  accomplish  this  without  putting  training  sites  at  risk  of  infection  from 
computer  viruses  will  require  further  investigation. 

A  final  common  issue  concerns  the  hardware  the  user  will  need  to  efficiently  run  the 
integrated  training  tool  application.  Experience  from  the  formative  evaluation  of  the  CITT 
showed  that  the  average  unit  will  probably  not  have  a  computer  powerful  enough  to  efficiently 
operate  a  large  training  information  management  program.  The  user  in  the  field  will  need  to  be 
equipped  with  an  adequate  computer.  Table  2  lists  recommended  basic  hardware  requirements 
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based  on  the  computers  being  purchased  for  the  West  Point  class  of  2002.  A  computer  meeting 
these  specifications  could  be  purchased  today  for  less  than  $3000.  Ideally,  this  computer  could 
be  purchased  for  each  user  at  the  time  an  implementation  decision  is  made. 

Table  2.  Recommended  System  Configuration  for  Fielding  an  Integrated  Training  Tool. 


Component 

Chip  Set 

RAM  Access  Memory 
Hard  Drive 

Compact  Disk-Read  Only  Memory 
Video  Card 

Sound  Card 
Modem 
Screen  Display 
Operating  System 
Browser 

Other  Applications 


Specification 
Pentium  450 
128  Megabyte 
12  Gigabyte 
DVD  II  ROM  Drive 

Video  Graphics  Adapter  with  4  Megabyte 
Video  RAM 

Sound  Blaster  Compatible  with  Speakers 
56.6  Kilobits  per  second 
17"  Monitor,  800  x  600  resolution 
NT  Workstation  4.0  with  Service  Pack  3 
Internet  Capable 

Word  Processing,  Spreadsheet,  and 
Graphics 


In  summary,  all  three  alternatives  for  implementing  an  integrated  training  tool  would 
allow  the  user  to  develop  training  materials  for  use  in  CCTT  and  all  three  share  common  fielding 
issues.  The  primary  difference  between  them  is  in  their  different  level  of  interaction  with  other 
Army  training  systems.  Only  the  third  alternative  involves  complete  integration  to  ensure 
synchronization  with  existing  systems  and  would  reduce  the  number  of  new  systems  the  user 
would  need  to  learn  to  use. 


PROJECT  EVALUATION  ACTIVITIES 

Evaluation  activities  for  the  Ci'lT  Project  needed  to  take  into  account  that  the  design  of 
the  CITT  is  the  primary  product  of  the  project.  As  described  in  Task  3,  the  CITT  design  is  not  a 
product  in  the  traditional  sense.  Rather,  the  design  is  a  conceptual  process  depicted  in  a  series  of 
IDEF  models,  ICOMs  and  FEOs,  Node  Trees,  and  Activity  Listings.  The  evaluation  of  the 
design  can,  at  least  partly,  be  accomplished  by  obtaining  the  opinions  of  individuals  with 
expertise  in  the  area  of  system  modeling,  but  ultimately  it  must  wait  for  implementation  in  order 
for  a  complete  evaluation  to  be  conducted.  On  the  other  hand,  the  project  did  produce  products 
in  the  form  of  the  CITT  videos  and  the  prototype.  Those  products  were  more  amenable  to 
traditional  evaluation  procedures.  In  addition,  the  project  itself  was  completed  in  the  context  of  a 
quality  assurance  model  which  looks  not  just  at  the  products,  but  also  at  the  activities  that 
produced  them.  According  to  Stebbing  (1986),  “the  ultimate  purpose  of  any  quality  assurance 


56 


scheme  is  to  ensure  complete  satisfaction  by  the  customer  with  the  goods  or  services  provided  by 
the  supplier...  the  real  evidence  of  quality  must  be  seen  to  exist,  not  only  in  the  completed  item 
but  in  all  activities  which  are  involved  in  completing  that  item...”  (page  8). 

The  project  team  established  review  procedures  to  assess  the  project  activities  and  to 
assure  that  the  project  remained  on  track  and  was  achieving  its  objectives.  The  primary 
assessment  of  the  CITT  design  was  through  this  review  process  as  was  the  assessment  of  the 
videos.  The  review  procedures  included  a  combination  of  team  meetings,  and  technical  reviews. 
The  primary  testable  product,  the  prototype,  was  assessed  through  a  series  of  evaluation 
activities  beginning  with  internal  software  testing  and  proceeding  through  formative  evaluation. 
The  remainder  of  this  section  will  report  on  the  CITT  Project  evaluation  activities  and  their 
results. 


Review  Process 

Early  in  the  project  life  cycle,  various  military  standards  were  researched  and  examined 
to  determine  their  applicability  to  this  project.  The  project  team  decided  to  adhere  to 
MIL-STD-498,  Software  Development  and  Documentation.  (DoD,  1994).22  The  selection  of 
MIL-STD-498  provided  the  project  team  with  a  standard  against  which  project  activities  and  the 
outcomes  of  those  activities  could  be  judged. 

At  the  heart  of  the  review  process  was  the  recognition  of  the  need  for  frequent  and 
detailed  communication  among  team  members  and  between  the  project  team  and  the  project 
sponsors.  Reviews  occurred  within  the  context  of  team  meetings,  technical  reviews,  COR 
reviews,  TSM  reviews,  and  In-process  reviews  (IPRs).  These  reviews  were  critical  to  the 
success  of  the  project. 


Team  Meetings 

In  the  early  stages  of  the  project,  team  meetings  were  held  frequently — at  least  weekly 
and  at  times  more  frequently.  These  frequent  meetings  provided  an  open  forum  for  discussion 
among  team  members  which  was  particularly  important  since  the  project  team  members  had 
become  somewhat  segregated  along  the  lines  of  Tasks  2-4.  It  was  important  that  everyone 
involved  knew,  at  least  at  a  general  level,  what  was  happening  in  all  phases  of  the  project.  The 
evaluation  specialist  kept  notes  for  the  team  meetings. 

It  was  also  at  team  meetings  that  the  team  made  important  decisions  related  to  the 
ongoing  conduct  of  the  project.  It  was  in  a  team  meeting,  for  example,  that  the  team  decided  to 
adhere  to  MIL-STD-498.  It  was  also  in  a  team  meeting  that  discussions  regarding  how  to  get 
modeling  back  on  track  occurred  when  that  process  started  to  bog  down.  Many  early  decisions 
regarding  the  design  and  development  of  the  prototype  occurred  at  team  meetings  as  did  early 
discussion  of  the  design  of  the  Instructional  Overview  and  videos. 


22  MIL-STD-498,  was  canceled  27  May  1998.  The  information  is  now  contained  in  Institute  of 
Electrical  and  Electronics  Engineers  (IEEE)/Electronics  Industries  Association  (EIA)  standard, 
EEEE/EIA  12207,  "Information  technology-Software  life  cycle  process".  This  change  had  no 
impact  on  the  evaluation  process  for  the  CITT  Project. 
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Although  at  times  there  was  considerable  disagreement  among  team  members,  team 
meetings  were  always  conducted  in  an  open  manner  where  everyone  felt  free  to  provide  his  or 
her  opinion,  and  where  everyone’s  participation  was  treated  with  respect.  This  was  the  key  to 
the  success  of  the  meetings. 


Technical  Reviews 


Technical  reviews  had  originally  been  planned  as  part  of  Task  4  and  were  intended  to 
provide  a  weekly  check  of  the  development  of  the  prototype.  In  actual  practice,  they  became  a 
more  formal  extension  of  the  team  meetings  and  all  areas  of  the  project  were  included.  These 
reviews  were  scheduled  to  occur  weekly  for  1 1  weeks  but  actually  ran  14  weeks.  The  team 
conducted  technical  reviews  very  similarly  to  the  team  meetings,  although  an  agenda  was 
typically  published  prior  to  the  meeting  and  was  generally  followed  during  the  meeting.  Because 
of  the  timing  of  the  technical  reviews,  the  prototype  development  usually  dominated  the  meeting, 
however,  as  indicated  above,  the  team  considered  all  phases  of  the  project. 

As  with  the  team  meetings,  the  technical  reviews  were  very  important  to  the  project  in 
terms  of  identifying  problems  early  and  determining  the  best  strategy  for  dealing  with  them.  In 
addition  to  problems  with  the  prototype,  other  problems  identified  included  difficulties  with 
video  production,  problems  with  COTS  software  that  had  been  selected  for  project  development, 
and  problems  with  the  design  tools  and  software. 

COR  and  TSM  Reviews 


To  obtain  feedback  from  the  COR  and  the  TSM  on  a  timely  basis,  frequent  meetings 
occurred  with  one  or  both  to  review  project  activities  and  products.  COR  reviews  were  generally 
informal  briefing  sessions  and  were  tied  to  major  project  decisions,  activities,  and  products.  For 
example,  the  COR  and  ACOR  received  briefings  on  and  reviewed  early  drafts  of  the  video 
scripts,  the  quality  assurance  plan  for  the  project,  early  design  considerations  for  the  CITT,  and 
prototype  specifications  among  others.  TSM  reviews  were  more  formal  and  were  generally  tied 
to  completion  of  major  products.  For  example,  the  TSM  reviewed  final  drafts  of  video  scripts, 
later  drafts  of  design  documents,  and  the  actual  working  prototype  among  others.  By  obtaining 
feedback  via  these  reviews,  the  project  team  could  address  concerns  of  the  project  sponsors  on  a 
timely  basis.  This  helped  to  ensure  that  the  project  would  meet  the  sponsor’s  needs  as  originally 
stated  in  the  SOW. 


In-process  Reviews 

A  total  of  four  In-process  Reviews  (IPRs)  occurred  during  completion  of  the  CITT 
Project.  Consistent  with  the  primary  goal  of  the  project,  the  IPRs  focused  on  the  design  of  the 
CITT.  Detailed  briefings  were  provided  at  each  IPR  and  feedback  was  obtained  from  the  COR, 
TSM,  PM-CATT,  ATSC,  proponent  schools,  and  other  government  representatives.  The  IPRs 
served  the  very  important  function  of  providing  information  to  and  obtaining  feedback  from  a 
relatively  large  group  of  individuals  representative  of  the  ultimate  sponsors,  providers,  and  users 
of  the  CITT. 
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The  critical  aspect  of  the  complete  review  process  employed  in  the  CITT  Project  was  that 
it  provided  for  frequent  feedback  to  the  project  team  from  an  ever-expanding  cadre  of  reviewers. 
The  frequency  of  the  reviews  needs  to  be  underscored.  Because  of  the  way  the  project  was  being 
completed,  individuals  or  small  groups  were  making  decisions  and  taking  actions  that  potentially 
had  large  impact  on  the  final  outcome  of  the  project  on  an  almost  daily  basis.  All  such  decisions 
and  actions  were  subject  to  review  within  several  days  at  most  either  within  the  project  team  or 
outside  the  team  as  necessary.  In  this  way,  the  likelihood  of  detecting  and  correcting  decisions 
and  actions  that  might  have  negative  impact  on  the  project  was  maximized  and  was  done  in  a 
very  timely  manner. 

One  final  point  needs  to  be  discussed  before  concluding  this  section.  As  stated  above,  the 
major  product  of  this  project  is  the  CITT  design — a  design  subject  to  detailed  review  throughout 
the  project  up  to  and  including  all  three  IPRs.  In  addition,  the  design  was  reviewed  by  subject 
matter  experts  in  information  systems  design  from  ATSC  who  reviewed  the  design  early  in  the 
project  and  then  again  as  the  design  was  further  refined.  Their  feedback  assisted  the  project  team 
in  better  understanding  some  of  the  more  complex  rules  of  activity  modeling  as  well  as  providing 
objective  feedback  on  the  logic  and  detail  of  the  CITT  design.  This  guidance  and  feedback 
assisted  the  project  team  in  refining  the  CITT  design  to  more  accurately  and  consistently 
illustrate  the  activity  and  function  of  the  application. 

Prototype  Evaluation 

The  prototype  was  subjected  to  a  multi-stage  evaluation  strategy  beginning  with  internal 
software  testing  conducted  by  members  of  the  project  team  and  continuing  through  three  levels 
of  user  testing  (User  Jury  testing,  informal  user  testing,  and  formative  evaluation.)  Keep  in 
mind,  however,  that  while  the  testing  process  can  be  described  as  a  linear  one,  the  actual  process 
is  very  interactive.  That  is,  internal  software  testing  for  one  unit  might  be  occurring  at  the  same 
time  user  testing  was  occurring  for  a  different  unit.  For  example,  user  testing  was  being 
conducted  on  the  Select  an  Exercise  and  Review  an  Exercise  modules  at  the  same  time  the 
Create  an  Exercise  module  was  undergoing  software  testing.  Furthermore,  the  results  of  testing 
may  well  impact  development  in  both  directions — units  that  are  further  along  in  development 
and  those  which  are  not  as  far  along.  So,  for  example,  it  was  not  uncommon  for  user  testing  of 
the  Select  an  Exercise  module  to  have  impact  on  the  Create  an  Exercise  module,  nor  was  it 
uncommon  for  software  testing  of  the  Create  an  Exercise  module  to  have  impact  on  the  Select  an 
Exercise  module  even  though  it  was  further  developed.  Due  to  its  earlier  completion  and  greater 
functionality,  more  testing  was  completed  on  the  CITTSA  than  on  the  Ci'iTDT  version.  The 
testing  of  the  CITTSA  will  be  described  and  differences  in  testing  of  the  CITTDT  will  be  noted. 

Internal  Software  Testing 

During  software  development,  the  team  employed  a  Software  Quality  Assurance  Program 
(SQAP)  to  detect  and  correct  as  many  defects  as  possible  as  early  as  possible.  The  SQAP 
consisted  of  unit  testing,  source-code  tracing,  integration  testing,  and  system  testing.  Defects 
identified  were  entered  into  a  defect-tracking  database  using  Census  97. 
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Software  unit23  testing  involves  informal  testing  of  source  code  by  the  developer  who 
wrote  the  code.  A  software  unit  can  refer  to  a  subroutine,  a  module,  or  even  the  complete 
program.  As  the  developer  completed  a  unit  of  code,  he  or  she  checked  it  for  logical  errors, 
syntax  errors,  etc.,  which  were  corrected  immediately.  Errors  found  and  corrected  at  this  level 
were  generally  not  entered  into  Census  97. 

Simultaneously  with,  or  immediately  following  software  unit  testing,  the  developer 
completed  source  code  tracing  using  an  interactive  debugging  program.  That  is,  the  code  was 
analyzed  step-by-step  using  a  program  designed  specifically  to  identify  coding  errors.  As  with 
unit  testing,  errors  were  corrected  immediately  and  were  generally  not  entered  into  Census  97. 
The  purpose  of  both  unit  testing  and  source  code  tracing  was  to  identify  and  correct 
programming  errors  prior  to  integration  of  the  code  which  greatly  reduced  integration  errors. 

After  software  units  had  been  tested  and  defects  corrected,  integration  testing  occurred  as 
units  were  brought  together  and  integrated  into  the  CITT.  In  integration  testing,  the  code  is 
exercised  as  a  complete  entity  to  determine  if  it  functions  as  needed.  As  newly  developed 
software  units  are  integrated  with  existing  software,  integration  testing  is  repeated.  Defects 
discovered  in  integration  testing  were  corrected  immediately,  if  possible,  in  which  case  they 
were  generally  not  entered  into  Census  97.  However,  if  a  defect  was  not  immediately 
correctable,  it  was  entered  into  Census  97  for  tracking  and  correction  later  in  development. 

The  final  stage  of  internal  software  testing  was  system  testing.  Unlike  unit,  source  code, 
and  integration  testing,  system  testing  was  conducted  by  other  members  of  the  development  team 
rather  than  by  the  developer  who  wrote  the  code.  This  was  done  to  preserve  objectivity.  For  the 
most  part,  the  project  manager  and  the  evaluation  specialist  performed  system  testing.  The 
testers  exercised  the  software  in  an  attempt  to  “break  it,”  and,  when  defects  or  errors  were  found, 
they  were  entered  into  Census  97.  System  testing  was  begun  when  the  Select  an  Exercise  and 
Modify  an  Exercise  modules  were  complete  and  continued  throughout  the  remainder  of  the 
development  of  the  CITTSA.  All  defects  identified  as  a  result  of  system  testing  that  could  be 
fixed  in  the  prototype  were  fixed  and  were  marked  closed  in  Census  97.  Resolution  was  not 
possible  for  some  problems  in  the  prototype  primarily  due  to  limitations  in  the  development 
software.  These  were  referred  to  possible  future  development,  and,  if  appropriate,  were  referred 
to  the  CUT  designers  for  an  analysis  of  possible  impact  on  the  design.  Of  148  entries  in  Census 
97,  approximately  40%  resulted  from  system  testing  indicating  that  a  substantial  number  of 
problems  were  found  and  corrected  prior  to  user  testing. 

Internal  software  testing  for  the  CITTDT  followed  a  different  process  primarily  because 
of  the  nature  of  the  development  process  itself.  The  CITTDT  is  not  an  application  in  the  same 
sense  as  the  CITTSA,  that  is,  it  is  not  an  integrated  software  program.  Instead,  it  is  a  collection 
of  active  server  pages  that  run  across  an  Internet  connection.  Each  page  is  built  and  tested  as  a 
separate  entity.  As  a  page  is  being  developed,  the  developers  get  immediate  feedback  on 
whether  or  not  the  code  behind  the  page  is  working  as  intended.  If  it  does  not,  modifications  are 
made  immediately  and  the  page  is  tested  again.  This  iterative  process  of  development  and  test 


23  Unit,  in  the  context  of  software  testing,  is  an  industry-wide  accepted  term  and  refers  to  testing 
of  software  code.  It  will  be  used  throughout  this  discussion  and  should  not  be  confused  with  a 
military  unit. 
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continues  until  the  page  works.  System  testing  did  occur  in  that  non-developers  accessed  the 
CITTDT  and  attempted  to  “break”  it  much  as  they  had  for  the  CITTSA.  Problems  and  defects 
uncovered  through  system  testing  were  entered  into  Census  97  and  were  provided  to  the 
CITTDT  developers. 


User  Testing 

User  testing  is  being  employed  broadly  to  cover  all  testing  involving  test  participants 
from  outside  the  development  team.  The  overarching  concern  of  all  levels  of  user  testing  was 
improvement  of  the  CITT  prototype.  The  purpose  of  user  testing  was  not  simply  to  detect  and 
correct  errors  in  the  prototype  itself,  but  also  to  determine  how  users  responded  to  the  CITT  in 
terms  of  such  variables  as  ease  of  use,  ability  to  accomplish  specific  tasks,  value  of  the 
information  presented,  etc.  To  this  end,  data  were  collected  from  User  Jury  testing,  informal 
user  testing,  and  formative  evaluation. 

User  Jury  Testing 

As  described  earlier  under  Task  4,  the  government  identified  a  User  Jury  consisting  of  as 
many  as  seven  individuals,  primarily  military  personnel  representative  of  the  CITT  user 
population.  Early  in  the  design  of  the  CITT,  the  jury  was  used  for  the  purpose  of  assisting  the 
project  team  in  making  design  decisions  primarily  in  the  area  of  user  interface  and  screen 
appearance  issues.  This  was  the  intended  function  of  the  jury  as  described  in  the  Research 
Program  Plan.  However,  in  later  sessions,  as  the  prototype  was  being  developed,  the  jury  was 
asked  to  provide  reactions  to  the  CITTSA  as  it  was  actually  being  implemented.  They  were 
provided  with  either  screen  shots  of  the  prototype  or  the  prototype  was  running  on  a  test 
machine,  and  the  jury  members  were  asked  for  their  reactions  regarding  how  the  CITTSA  was 
being  implemented.  Out  of  a  total  of  six  meetings  of  the  jury,  the  last  three  were  at  least 
partially  concerned  with  obtaining  the  jury’s  reactions  to  the  prototype. 

Based  on  User  Jury  feedback,  a  number  of  modifications  occurred  to  the  prototype. 
Recommendations  made  and  followed  regarding  general  issues  included  naming  conventions  for 
buttons  that  were  used  on-screen,  the  location  of  buttons  on  the  toolbar  and  on-screen,  and  the 
need  for  consistency  in  navigation,  as  well  as  specific  comments  related  to  individual  screens. 

In  general,  the  jury  provided  valuable  information  to  the  development  team,  and,  on 
balance,  was  a  productive  use  of  resources.  However,  the  overall  value  of  the  jury  was  probably 
limited  due  to  a  number  of  factors.  It  was  very  difficult  to  get  all  members  together  at  the  same 
time.  Several  meetings  were  conducted  with  as  few  as  four  of  the  members  present.  Meetings 
were  deliberately  kept  to  90  minutes  which  sometimes  was  not  enough  time  to  allow  for 
sufficient  discussion  of  issues.  And,  the  project  team  broadened  the  scope  of  the  jury  to  include 
issues  related  to  CITT  design  beyond  the  original  intent.  This  resulted  in  less  time  being  spent 
on  user  interface  and  system  usability  issues. 

Informal  User  Testing 

To  distinguish  this  testing  from  that  planned  for  formative  evaluation,  the  team  used  the 
term  “informal  user  testing.”  Informal  refers  to  the  test  atmosphere  rather  than  to  the  manner  in 
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which  testing  was  conducted;  sessions  were  conducted  less  formally  than  those  in  formative 
evaluation,  and  there  was  more  interaction  between  the  test  participants  and  the  observers. 
Informal  user  testing  began  on  21  July  1998  and  continued  through  20  August.  Fourteen  test 
sessions  were  conducted  for  a  total  of  34  hours  of  testing.  Test  cases  consisting  of  directions  to 
the  participants  regarding  the  tasks  they  were  to  complete  and  the  outcomes  they  were  to  produce 
were  created  for  each  session  as  per  the  Research  Program  Plan.  Participants  for  the  initial 
sessions  were  civilian  government  employees  who  were  given  fairly  broad  cases,  e.g.,  they  were 
instructed  simply  to  examine  CUT  and  see  if  they  could  “break”  it.  Later  sessions  were  ran  with 
military  personnel  and  involved  more  specific  test  cases,  e.g.,  “access  CITT  and  go  through 
Learn  About  CCTT.”  Two  members  of  the  project  team  observed  each  test  session  and  recorded 
observation  data  on  pre-printed  forms.  The  team  determined  early  in  informal  user  testing  that 
two  observers  were  required  and  that  at  least  one  of  them  needed  to  have  expertise  in  structured 
training  and  TSPs.  This  finding  proved  important  in  later  planning  for  formative  evaluation. 

The  observation  procedure  employed  for  informal  user  testing  is  best  termed  “active.” 
That  is,  the  observers  offered  assistance  to  the  test  participant  if  they  thought  he  or  she  was 
having  difficulty,  or  they  probed  for  additional  information  from  the  participant  when  an  action 
he  or  she  had  taken  was  unclear.  Active  observation  was  appropriate  for  informal  user  testing 
since  its  primary  function  was  to  find  and  correct  problems  with  the  CITTSA  as  opposed  to 
determining  if  the  CITTSA  supported  the  user  in  selecting,  modifying  or  creating  an  exercise. 
The  latter  was  the  function  of  formative  evaluation.  Observers  recorded  actions  taken  by  the 
participant,  problems  the  participant  had  completing  an  action,  problems  with  the  CITT  itself 
(e.g.,  buttons  not  working,  incorrect  links,  forms  working  improperly,  etc.),  and  questions  and 
comments  the  participant  had.  Following  each  test  session,  a  summary  report  was  prepared  for 
the  session. 

On  approximately  a  weekly  basis,  the  observers  met  to  analyze  the  test  results  from  the 
sessions  conducted  that  week.  The  summary  reports  served  as  the  basis  for  the  review  and 
discussion  during  these  meetings.  Based  on  the  analysis,  the  observers  listed  problems  with  the 
CITT  and/or  enhancements  to  the  CITT  that  needed  to  be  made  in  the  prototype.  They  also 
decided  what  changes/modifications  could  not  be  made  in  the  prototype  and  whether  they  would 
be  incorporated  into  the  CITT  design.  Changes/enhancements  to  the  prototype  were  entered  into 
Census  97;  implications  for  design  were  given  to  the  CITT  designers.  Thirty  to  forty  percent  of 
the  items  in  the  Census  97  database  resulted  from  informal  user  testing. 

Formative  Evaluation 


The  CITT  SOW  specifically  called  for  the  refinement  of  the  prototype  CITT  through 
formative  evaluation  (FE).  This  was  addressed  in  the  Research  Program  Plan  under  Beta  testing, 
and  this  terminology  was  used  in  the  early  months  of  the  project.  On  further  analysis,  however, 
the  team  decided  that  formative  evaluation  is  the  more  precise  term  to  describe  what  was  planned 
and  what  actually  occurred,  and  that  terminology  will  be  used  for  the  remainder  of  the  report. 
Formative  evaluation,  as  described  by  Scriven  (1991)  is  evaluation  typically  conducted  during 
development  of  a  product  with  the  intent  to  improve  the  product. 
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Formative  evaluation  plan.  In  February,  1998,  a  plan  for  completing  FE  was  presented 
to  the  COR.  The  plan  addressed  three  primary  objectives  taken  directly  from  the  SOW. 
Specifically,  FE  was  to  assess  the  ability  of  commanders  and  other  unit  trainers  to: 

1 .  Select,  modify  and  develop  CCTT  exercises  for  platoons  and  company  teams. 

2.  Access,  modify,  develop  and  print  required  training  support  materials. 

3.  Execute  fully  the  exercises  as  selected,  modified  or  developed. 

The  first  two  objectives  are  “internal”  to  the  CITT,  that  is,  they  do  not  require  the  test 
participants  to  perform  any  actions  outside  of  those  included  in  or  supported  by  the  CITT.  The 
third  objective,  on  the  other  hand,  is  assessing  actions  or  functions  “external”  to  the  CITT,  that 
is,  can  exercises  selected,  modified,  or  created  be  fully  executed  at  the  CCTT  site.  This  means 
involving  additional  personnel  such  as  workstation  operators  and  site  personnel  and  requires  two 
additional  FE  objectives.  Specifically  FE  needed  to  determine  the  CITT’s  ability  to: 

4.  Provide  other  unit  personnel  with  the  capability  to  access  required  information  necessary  to 
support  the  execution  of  exercises  created  or  modified. 

5.  Provide  CCTT  site  personnel  with  the  capability  to  build  exercises  that  have  been  created  or 
modified. 

As  described  below,  due  to  scheduling  problems,  there  was  insufficient  time  to  build  the 
exercises  created  during  testing  limiting  FE  to  three  of  the  above  objectives  (the  first,  second  and 
fourth.) 

The  FE  plan  identified  several  constraints  that  impacted  data  collection: 

1.  Testing  was  to  be  conducted  using  members  of  the  target  population  for  whom  the  CITT  was 
being  developed,  specifically  soldiers  from  Fort  Hood  and  Fort  Knox. 

2.  Testing  was  to  be  balanced  between  the  two  prototype  versions. 

3.  Test  participants  were  to  be  drawn  from  both  active  and  reserve  (including  Army  National 
Guard)  components  of  the  Army. 

4.  Evaluation  data  were  to  be  collected  using  two  primary  methodologies: 

a.  Passive  observation  in  which  users  were  monitored  (both  by  the  CITT  team  and  by  the 
CITT  system)  as  they  used  the  system.  Observation  was  passive  in  that  the  observers 
would  not  offer  help  or  guidance  to  the  participant.  All  problems/deficiencies 
encountered  were  recorded  either  by  the  CITT  or  by  observers  from  the  CITT  team, 

b.  Surveys/interviews  of  users  involved  in  the  FE  were  conducted  by  the  observers. 

During  FE,  data  were  to  be  collected  by  monitoring  test  participants  as  they  completed  a 
test  case,  and  by  interviewing  and/or  administering  questionnaires  to  test  participants.  Test 
cases  (i.e.,  the  specific  scenario  for  using  CITT  which  a  participant  was  asked  to  follow)  were 
developed  to  fit  the  training  needs  of  the  Fort  Hood  units;  additional  test  cases  were  developed 
for  testing  at  Fort  Knox  in  order  to  exercise  to  the  maximum  extent  possible  all  of  the  CITT 
functionalities.  Passive  monitoring  was  employed  during  testing,  and  participants  completed  the 
embedded  questionnaire  (see  Appendix  D).  On  the  other  hand,  it  was  not  possible  to  include 
embedded  data  collection  tools  in  the  CITT  itself,  thus  this  method  was  not  employed.  The  Data 
to  be  collected  during  FE  included: 
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1 .  Time  to  complete  each  module. 

2.  Paths  taken  through  CITT  cross  referenced  to  test  case. 

3.  Successful  outcome  reached  (e.g.,  exercise  was  selected,  created,  modified;  participant  able 
to  access  EDUCCATT  to  utilize  workstation  training,  etc.). 

4.  Navigation  errors  (i.e.,  participant  had  to  “back  up”  to  get  back  on  task;  participant  went 
down  a  path  that  did  not  lead  to  desired  outcome). 

5.  System  errors  (e.g.,  links  not  working  correctly,  buttons  not  working,  help  screens  not 
working,  etc.). 

6.  Usability  measures  (e.g.,  “look  and  feel”,  ease  of  navigation,  ease  of  use,  etc.), 

7.  Functions  and  features  omitted. 

8.  Desired  system  changes. 

9.  Difficulties  encountered/errors  made  during  exercise  build. 

10.  Difficulties  encountered/errors  made  during  exercise  execution. 

Finally,  a  timeline  was  included  with  the  plan  which  specified  identification  of  the  test 
unit  four  months  prior  to  its  CCTT  training.  This  allowed  for  testing  of  the  CITT  prototype  one- 
and-a-half  to  two  months  prior  to  CCTT  training,  thus  providing  the  CCTT  site  personnel 
sufficient  time  to  build  and  proof  the  exercises  created  or  modified  using  CITT.  When  the  unit 
trained  at  the  CCTT  site,  they  would  train  on  the  modified/created  exercises,  thus  allowing 
testing  of  objective  3.  This  schedule  was  consistent  with  CCTT  site  requirements. 

Approximately  two  weeks  for  CITT  testing  would  be  required  at  Fort  Hood  and  one  week 
at  Fort  Knox.  In  addition,  a  week  at  Fort  Hood  would  be  required  to  build/proof  the  exercises, 
and  two  weeks  would  be  required  to  test  whether  the  exercises  were  fully  executable  by  the  unit. 

Implementation  of  the  plan.  For  a  variety  of  reasons,  the  actual  FE  of  the  prototype 
varied  from  the  plan  in  a  number  of  important  ways.  First,  units  were  not  identified  until  August. 
In  early  August,  the  COR  and  project  manager  visited  the  squadron  at  Fort  Hood  to  discuss  the 
testing  schedule.  Units  were  scheduled  for  CCTT  training  the  weeks  of  28  September  and 
5  October,  and  the  only  time  available  for  testing  the  CITT  was  the  week  of  14  September.  This 
had  two  major  effects.  The  amount  of  testing  at  Fort  Hood  had  to  be  reduced  and  the  amount  of 
testing  at  Fort  Knox  had  to  be  increased.  More  importantly,  from  a  testing  standpoint,  there  were 
only  two  weeks  at  most  for  the  CCTT  site  personnel  to  build  and  proof  exercises  which  was 
insufficient.  The  build/proof  test  and  the  execute  exercises  test  were  abandoned.  As  stated 
above,  this  limited  FE  to  a  test  of  three  of  the  five  objectives. 

Another  variance  from  the  plan  concerned  the  testing  of  the  CITTSA  versus  the  Ci'lTDT. 
As  described  under  Task  4,  development  of  the  CITTDT  lagged  significantly  behind 
development  of  the  CITTSA.  Because  of  this,  the  project  team  decided  that  all  testing  at  Fort 
Hood  and  the  majority  of  testing  at  Fort  Knox  would  be  conducted  on  the  CITTSA.  Testing  of 
the  CITTDT  was  limited  primarily  to  the  Learn  About  CITT  and  Select  and  Review  Exercise 
modules. 

Following  negotiations  with  the  test  units,  the  testing  schedules  for  Fort  Hood  and  Fort  Knox 
changed,  and  these  are  shown  in  Figure  3 1 .  For  FE  conducted  at  Fort  Hood,  the  majority  of 
testing  was  completed  with  the  two  troop  commanders.  From  previous  discussion,  the  project 
team  knew  that  one  troop  commander  would  be  modifying  existing  exercises,  and  the  other 
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would  be  creating  new  exercises.  Their  test  cases  were  based  on  this  and  adapted  to  their  needs. 
In  addition,  the  team  decided  to  test  the  O/C,  FDC,  and  CTCP  workstation  operator  functions  of 
the  CUT  and  requested  personnel  from  the  squadron  appropriate  for  those  tests.  Test  cases  were 
prepared  for  the  workstation  operators  which  directed  the  participant  to  use  CITT  to  obtain  and 
examine  all  of  the  information  they  would  need  to  support  the  exercise(s)  their  unit  would  be 
using  in  CCTT.  During  testing,  however,  the  team  learned  that  the  commanders  would  be 
observing  their  own  exercises,  thus,  the  O/C  case  was  dropped  from  the  test.  The  Contractor 
Logistics  Support  (CLS)  site  person  was  given  a  very  general  case — learn  about  the  CITT 

The  Fort  Knox  test  schedule  changed  to  add  additional  participants  to  compensate  for 
participants  who  were  dropped  from  the  Fort  Hood  schedule.  The  ARI  Research  Coordinator 
contacted  and  coordinated  the  scheduling  of  individuals  from  Fort  Knox  who  would  serve  as  test 
participants. 
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Figure  3 1 .  The  modified  FE  plan  for  Fort  Hood  and  Fort  Knox. 

The  project  team  had  learned  during  informal  user  testing  that  two  observers  were 
required  for  each  test  session,  and  that  at  least  one  of  them  needed  expertise  in  structured  training 
and  TSPs.  FE  observers  were  selected  who  fit  these  criteria.  Since  EE  at  Fort  Hood  was 
occurring  first,  and  since  two  test  machines  were  being  used,  a  total  of  four  observers  were 
required.  The  project  manager,  the  evaluation  specialist,  a  project  team  subject  matter  expert 
(SME),  and  an  SME  from  the  STRUCCTT-2  team  were  selected  as  observers. 
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Data  collection  and  observation  forms  were  developed  for  each  of  the  three  major  testing 
activities  (test  CITT,  test  the  build/proof  process,  and  test  the  exercise  execution.)24  Two  types  of 
forms  were  developed  for  each  session.  (See  Appendix  E  for  examples  of  all  data  collection 
forms.)  Observation  forms  were  developed  to  be  used  by  the  observers  during  the  CITT  test, 
during  build/proof,  and  during  exercise  execution.  For  the  CITT  test  forms,  all  of  the  actions  a 
user  could  take  in  the  CITT  were  listed  and  a  scheme  for  abbreviating  them  was  developed.  This 
facilitated  observation.  In  addition,  test  assistance  guidance  was  developed  which  provided 
options  for  the  observer  to  assist  the  participant  during  testing.  Consistent  with  the  directive 
from  the  SOW  to  use  passive  observation,  observers  received  instruction  to  provide  the 
minimum  assistance  necessary.  The  first  option  was  to  suggest  the  participant  try  the  built-in 
CITT  help.  If  that  failed,  observers  were  told  to  provide  a  general  hint  or  assistance  such  as, 

“you  might  want  to  consider  the  task  steps  you  want  the  unit  to  perform.”  If  assistance  was  still 
needed,  they  were  told  to  provide  a  direct  instruction  such  as,  “Go  to  Initial  Settings  and  list  the 
task  steps  you  want  the  unit  to  perform.”  Although  providing  assistance  goes  beyond  passive 
observation,  the  project  team  decided  the  alternative  (allowing  the  participant  to  get  further  and 
further  “lost”)  would  be  unacceptable.  This  approach  to  providing  assistance  worked  well. 

The  abbreviation  code  and  the  assistance  guidance  appeared  on  each  left  hand  page  of  the 
observation  booklets  to  be  used  as  an  aid  to  the  observer  during  testing.  In  addition, 
interview/debriefing  forms  were  developed  for  each  test  session  using  questions  germaine  to  the 
specific  test.  These  questions  appeared  on  the  left  hand  pages  of  the  recording  booklets  to 
facilitate  the  interviews.  In  the  actual  FE,  only  the  CITT  Test  observation  and 
Interview/debriefing  booklets  were  used. 

On  10  September,  a  familiarization/training  session  was  conducted  for  the  test  observers. 
The  purpose  of  the  FE  testing  was  discussed  at  length,  and  observation  and  interview/debriefing 
procedures  were  covered. 

On  14  September,  the  first  test  sessions  were  conducted  with  the  troop  commanders  at 
Fort  Hood.  The  FE  plan  specified  that  the  commanders  would  participate  in  the  test  for 
approximately  two  and  a  half  days  and  would  be  interviewed  the  afternoon  of  the  third  day.  It 
was  anticipated  that  both  commanders  would  develop  (either  by  modifying  or  creating)  several 
exercises.  In  actuality,  this  did  not  occur.  One  commander  participated  for  a  total  of  15  hours 
excluding  the  interview/debrief.  The  second  commander  participated  for  approximately  10  hours 
excluding  interview/debrief.  Both  had  other  duties  and  responsibilities  with  their  troops  which 
prevented  them  from  participating  for  the  entire  day.  In  fact,  one  was  unable  to  participate  at  all 
on  the  16th,  however,  he  did  return  on  the  17th  and  completed  the  exercise  he  had  started. 
Following  their  test  on  the  CITT,  each  commander  was  requested  to  complete  the  CITT 
questionnaire  and  was  then  asked  to  participate  in  the  interview/debriefing  which  took 
approximately  an  hour. 

On  the  17th,  FABTOC  and  FDC  workstation  operators  were  tested.  However,  the 
commander  who  had  tasked  them  to  participate,  sent  the  participants  in  pairs  which  made 


24  Data  collection  and  observation  forms  were  developed  prior  to  the  decision  that  the  build/proof 
and  exercise  execution  components  of  FE  would  have  to  be  dropped.  Discussion  of  them  is 
included  in  the  report  for  completeness. 
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interpretation  of  the  session  somewhat  difficult.  They  were  able  to  complete  the  test  cases, 
however,  post-test  debriefing  occurred  only  for  one  of  the  pairs.  Also  on  the  17th,  the  Contractor 
Logistics  Support  (CLS)  site  contractor  was  tested,  although  his  test  case  was  very  general.  He 
was  told  simply  to  familiarize  himself  with  CITT. 

In  general,  it  is  fair  to  say  that,  with  the  exception  of  a  few  minor  problems  and  the 
necessity  to  adapt  to  the  schedules  of  the  soldiers  participating  in  the  test,  the  FE  sessions  at  Fort 
Hood  went  well.  However,  one  additional  point  should  be  noted  regarding  testing  at  Fort  Hood. 
Although  no  test  participant  was  asked  to  use  the  CITTDT,  it  was  tested  by  members  of  the 
project  team  and  by  the  TSM.  The  CITTDT  was  accessed  using  a  network  connection  that  went 
through  the  post  local  area  network  (LAN).  This  was  done  to  determine  how  it  would  operate 
under  conditions  that  represent  those  typically  found  on  an  Army  installation.  This  testing  was 
very  informal  and  observation  data  were  not  collected.  Essentially,  it  was  a  “let’s  see  what 
happens”  test  the  results  of  which  will  be  reported  below. 

FE  testing  at  Fort  Knox  began  on  the  21st  of  September  and  continued  through  the  6th  of 
October.  The  ARI  Research  Coordinator  had  arranged  a  testing  schedule  that  involved  one 
three-hour  test  session  per  day  and  covered  all  of  the  test  cases  shown  in  Figure  31.  A  total  of  10 
participants  were  tested — eight  for  one  session  each  and  two  for  two  sessions  each.  The 
participants  tested  for  two  sessions  were  the  Battalion  S3  and  the  CLS  contractor.  Test  cases 
were  prepared  in  advance  for  all  sessions.  One  participant  received  instructions  to  use  the 
CITTDT;  all  other  test  cases  involved  the  CITTSA.  (The  Brigade  Commander  was  given  a  very 
general  test  case  and  actually  used  both  versions.) 

Sessions  were  observed  by  the  evaluation  specialist  and  the  project  team  SME  who  had 
been  an  observer  at  Fort  Hood.  (The  evaluation  specialist  was  unable  to  observe  one  session  and 
was  replaced  by  the  CITTSA  database  application  developer.)  Observations  were  recorded  in 
the  CITT  Test  Observation  Booklets  and  interview/debriefing  results  were  recorded  in  the  CITT 
Test  Debriefing  Booklet.  In  addition,  most  participants  were  requested  to  complete  the  CITT 
questionnaire. 

As  at  Fort  Hood,  there  were  no  major  problems  with  FE  testing  at  Fort  Knox.  With  one 
exception,  all  participants  arrived  as  scheduled,  and  all  sessions  were  conducted  as  planned. 

Formative  evaluation  results.  Findings  from  formative  evaluation  will  be  described  in 
two  categories:  findings  representing  problems/defects  or  shortcomings  in  the  CITT  itself  (e.g., 
coding  errors,  incorrect  links,  functions  and  features  not  working,  etc.),  and  findings  which 
reflect  the  test  participant’s  reactions  to  the  CITT  in  terms  of  usability,  appearance,  user 
interface,  etc. 

Following  the  completion  of  all  test  cases,  the  observers  met  to  analyze  the  test  results. 
The  initial  analysis  involved  reviewing  all  of  the  observation  data  to  determine  problems/defects 
or  shortcomings  in  the  CITT.  This  analysis  resulted  in  two  lists  of  defects:  those  to  be  corrected 
in  the  prototype  and  those  to  be  assigned  to  future  updates/functionalities.  A  list  of  29  prototype 
defects  or  shortcomings  was  identified  and  is  included  as  Appendix  F — CITT  Improvements 
Implemented.  This  list  was  given  to  the  project  team  developers  and  was  used  to  refine  the  CITT 
prototype.  A  second  list  of  defects/enhancements  or  shortcomings,  consisting  of  33  items,  was 
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identified  from  the  FE  data  and  is  included  in  Appendix  G — CUT  Improvements  To  Be 
Implemented.  This  list  represents  desired  functionalities  for  a  future  CITT  that  were  not 
implemented  in  the  prototype.  Some  of  these  desired  functionalities  are  included  in  the  CITT 
design,  i.e.,  they  were  considered  when  CITT  was  designed  but  were  unable  to  be  implemented 
in  the  prototype.  Others  represent  modifications  and  enhancements  to  the  design. 

The  FE  data  were  also  analyzed  for  user  reactions  to  the  CITT  using  survey  data 
collected  from  the  CITT  questionnaire  and  data  from  the  interview/debriefing  sessions.  Survey 
data  were  analyzed  quantitatively  and  qualitatively;  the  remaining  data  were  analyzed 
qualitatively  because  they  were  actually  more  like  individual  case  studies  rather  than  group  data. 
There  were  relatively  few  participants  (only  10  from  whom  data  were  collected  individually). 
And,  each  participant  was  tested  using  a  unique  test  case.  This  meant  that  each  participant  was 
completing  a  different  set  of  activities,  albeit  with  a  great  deal  of  commonality.  This  greatly 
limits  the  generalizability  of  the  data  collected. 

Fifteen  participants  completed  the  questionnaire.25  Nine  participants  rated  themselves  as 
very  or  moderately  experienced  with  using  personal  computers.  In  response  to  “How  well  were 
you  able  to  accomplish  what  you  wanted  to  do?”  over  half  (8)  of  the  participants  said  well  or 
very  well.  Six  participants  said  they  were  able  to  accomplish  what  they  want  to  a  fair  amount, 
and  one  said  not  at  all.  When  asked  how  difficult  it  is  to  navigate  through  the  CITT,  12 
participants  responded  either  very  easy  or  easy.  No  participant  thought  it  was  extremely 
difficult.  Thirteen  participants  thought  navigation  tools  and  buttons  were  used  consistently  or 
very  consistently  throughout  the  CITT,  and  1 1  indicated  that  it  was  easy  or  very  easy  to  get  back 
to  where  they  wanted  to  be  if  they  became  “lost”. 

Ten  of  the  participants  indicated  that  they  had  used  the  help  feature  of  the  CITT  during 
their  session,  the  majority  of  them  had  used  help  ten  times  or  less.  One  participant  had  used  help 
more  than  20  times.  Of  these  10  participants,  eight  said  the  built-in  Help  material  were  either 
very  helpful  or  somewhat  helpful.  Seven  participants  thought  the  Help  material  was  written  at 
the  correct  level,  and  only  one  participant  said  he  or  she  was  unable  to  obtain  Help  when  they 
needed  it.  In  explanation,  that  participant  said  he  could  not  obtain  help  at  the  start  of  the  session. 

In  response  to  “What  feature  of  the  CITT  did  you  find  most  useful?”  a  variety  of 
responses  appeared.  Two  participants  mentioned  the  linkages  that  are  built  into  the  CITT.  Other 
items  mentioned  included  the  Matrix  OPORD,  the  Audio  feature,  the  mission  layouts  and 
thumbnails,  the  help  fields,  and  the  graphic  software.  In  response  to  the  least  useful  features 
item,  again  a  variety  of  responses  appeared  including  filling  in  S  AF  for  BLUFOR  units,  the 
Access  database  and  TSP  approach,  structured  training,  buttons  not  working,  the  problem  of 
marking  an  exercise  complete  and  then  later  wanting  to  change  it,  and  the  need  to  have  the 
graphic  display  files  change  as  the  terrain  associated  with  the  graphic  overlays  changes.  Caution 


The  apparent  discrepancy  between  15  participants  completing  the  questionnaire  and  12 
participants  from  whom  data  were  collected  individually  is  easily  explained.  One  participant 
who  participated  in  more  than  one  test  session  completed  the  questionnaire  more  than  once. 
Also,  four  of  the  workstation  operators  who  had  tested  at  Fort  Hood  in  pairs  each  completed  the 
questionnaire  individually.  Two  of  the  Fort  Knox  participants  did  not  complete  the 
questionnaire. 
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should  be  exercised  in  interpreting  these  results,  however,  since,  for  the  most  part,  they  represent 
the  response  of  a  single  participant. 

When  asked  what  features  the  participants  would  like  to  see  added  to  the  CITT,  only 
eight  participants  responded  and  there  was  no  consistency  among  responses.  Responses  included 
adding  scanning  in  graphics  and  the  ability  to  click  on  a  map  and  have  the  system  automatically 
assign  locations,  removing  the  wizard,  the  ability  to  print  selected  parts  of  the  TSP,  crew  level 
training,  step-by-step  instructions,  and  adding  a  password  option  to  allow  access  to  an  exercise 
even  after  it  had  been  marked  complete. 

Interview/debriefing  data  showed  the  same  wide  range  of  responses  that  survey  data  did 
and  generally  support  the  survey  data.  Generally  speaking,  overall  reaction  to  the  CITT  was 
very  positive.  Only  one  participant  was  unable  to  complete  the  test  case  which,  in  itself,  is  a 
very  positive  indication  of  the  system’s  usability.  CITT  is  a  relatively  complex  system  and  the 
fact  that  almost  all  participants  were  able  to  use  it  satisfactorily  on  their  first  attempt  is 
significant. 

In  general,  all  participants  had  an  overall  positive  impression  of  the  CITT.  They  thought 
it  was  relatively  easy  to  use  and  relatively  easy  to  navigate.  A  number  mentioned  the  different 
ways  to  navigate  as  a  plus.  The  Wizard  Agent  was  probably  the  only  feature  that  was  viewed 
with  ambiguity.  Participants  either  liked  it  a  lot  or  thought  it  was  a  nuisance.  The  10  was 
mentioned  several  times  as  a  positive  feature;  participants  generally  liked  the  tree-view  feature. 
Several  also  mentioned  the  audio-visuals  that  are  included  in  the  10. 

Some  representative  comments  from  the  interview/debriefing  session  are  shown  in 
Table  3. 

Table  3.  Comments  from  interview/debriefing _ _ _ 

This  is  far  more  than  I  expected  from  an  Army  Program.  (Said  of  the  Cl’ITSA) 

As  far  as  Army  Web  sites,  it  was  great.  (Said  of  the  COTDT) 

The  CITT  has  the  potential  for  all  types  of  training,  not  just  for  CCTT. 

Good  system,  user  friendly,  but  time  consuming. 

It  is  a  complex  process  with  levels  within  levels.  This  is  sometimes  confusing. 

It  is  easy  to  miss  the  Tabs. 

I  was  able  to  find  everything  I  needed;  it’s  relatively  complete. 

You  need  a  way  to  keep  track  of  where  you  are. 

Need  to  make  the  process  of  finding  an  exercise  simpler. 

It  needs  a  tutorial  on  using  a  Windows  environment. 

Unit  commanders  won’t  have  time  to  use  CITT. 


As  Table  3  indicates,  the  CITT  was  successful  in  achieving  the  objectives  specified; 
however,  considerable  room  for  improvement  still  remains. 

Finally,  a  word  about  the  results  of  the  informal  test  of  the  COTDT  from  Fort  Hood 
using  the  post  LAN.  Because  every  user  has  direct  access  to  the  Internet  rather  than  going 
through  a  proxy  server,  traffic  becomes  very  heavy  during  normal  working  hours.  It  was  not 
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uncommon  for  a  page  to  require  4-5  minutes  to  load,  and  some  pages  required  as  much  as  10 
minutes.  This  is  deemed  unacceptable  for  typical  users. 


LESSONS  LEARNED 

The  CITT  Project  represents  a  “first  approach”  to  providing  soldiers  with  a  viable 
training  support  package  (TSP)  generator  and  information  system  using  a  state-of-the-art 
simulation  system.  Throughout  the  project,  a  number  of  lessons  were  learned  from  this  effort 
that  will  be  instrumental  in  future  endeavors  involving  systems  of  this  nature.  This  section 
describes  lessons  learned  for  the  project  in  general  and  for  design  and  prototype  development 
and  testing. 


General 

As  the  project  team  was  designing  CITT,  and  particularly  while  developing  the  prototype, 
the  team  realized  that  many  of  the  tasks  that  can  be  simulated  in  the  COT  cannot  actually  be 
executed  realistically,  and  thus  have  unknown  value  as  far  as  how  well  the  training  will  transfer 
to  actual  task  performance.  It  is  imperative,  therefore,  that  the  Army  define  task  performance 
support  codes  (TPSCs)  for  all  MTPs  and  all  simulations.  Given  the  limited  training  time 
available  to  units,  and  as  they  rely  more  and  more  on  virtual  training  environments,  it  becomes 
particularly  important  that  the  units  concentrate  on  those  tasks  which  have  the  greatest 
transference.  The  unit  in  the  field  is  being  misled  and  can  waste  valuable  time  and  energy  on 
designing  training  for  tasks  that  can  not  be  executed  on  the  selected  simulation. 

The  Army  must  examine/determine  the  TSP  components  and  formats  needed  to  support 
training  at  all  echelons  and  for  all  environments  (virtual,  constructive,  and  live).  Currently, 

SATS  provides  a  template  for  TSPs,  however  it  is  too  generic  to  apply  to  all  of  the  training 
situations  discussed  in  the  fielding  plan.  The  SATS  TSP  components  can  serve  as  the  core 
requirements,  however,  additional  components  necessary  to  support  the  different  types  of 
training  and  exercises  envisioned  need  to  be  identified.  This  is  especially  important  if  an  attempt 
is  made  to  develop  a  “master”  training  data  base  as  envisioned  in  the  fielding  plan  that  will 
support  training  at  all  echelons  and  in  all  environments.  One  way  to  identify  all  of  the  elements 
and  relationships  needed  in  this  database  is  by  examining  the  TSP  components  needed  for 
different  types  of  training/exercises  and  reverse  engineering  the  database  requirements  from 
them. 


CITT  Design 

It  is  extremely  important  in  modeling  to  identify  the  point  of  view  (POV)  from  which 
modeling  is  being  completed  and  to  model  from  that  perspective.  This  may  sound  obvious,  but 
in  point  of  fact,  is  often  quite  difficult  to  accomplish.  In  die  CITT  Project,  for  example,  even 
though  the  initial  POV  for  modeling  was  the  Commander/Unit  Trainer,  the  project  team  often 
slipped  into  modeling  from  the  standpoint  of  the  capabilities  of  the  simulator  or  from  the 
structure  and  format  of  the  TSP.  This  resulted  from  the  designers’  familiarity  with  and 
knowledge  about  the  CCTT  and  the  TSP  format  developed  for  the  STRUCCTT  projects,  and  it 
was  sometimes  easy  to  think  in  those  terms.  Modeling  from  the  POV  of  the  Commander/Unit 
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Trainer  required  a  conscious  decision-making  process  that  continuously  asked  “what  will  the 
user  be  doing  with  this?”  As  modeling  is  completed  for  other  users,  the  same  decision-making 
process  is  required. 

The  project  team  underestimated  the  time  required  to  leam  to  do  modeling.  Even  though 
there  are  numerous  references  and  standards  that  describe  the  modeling  process,  there  is  still  a 
substantial  learning  curve.  The  process  itself  is  complex.  Future  projects  must  either  allow 
adequate  time  to  leam  to  model  or  should  make  sure  that  at  least  one  member  of  the  project  team 
has  the  requisite  modeling  skills. 

The  CITT  design,  particularly  the  initial  modeling  efforts,  proved  to  be  difficult  because 
of  problems  the  team  had  separating  design  from  prototype  development.  Modeling  frequently 
became  bogged  down  with  discussions  of  whether  the  prototype  development  software  would 
work  a  certain  way,  or  whether  a  given  activity  would  be  able  to  be  implemented  in  the 
distributed  version,  etc.  This  was  true  even  though  the  team  understood  and  agreed  that  the 
primary  goal  of  the  project  was  to  design  a  CITT,  not  to  develop  a  prototype.  The  difficulty  is 
that  modeling  is  a  very  abstract  process  while  prototype  design  and  development  are  very 
concrete,  and  it  is  easy  for  the  concrete  to  overwhelm  the  abstract.  The  solution  was  to  develop  a 
design  and  modeling  process  that  clearly  separated  CITT  design  from  prototype  development. 
The  process  consisted  of  the  following  basic  steps:  an  activity  was  identified  and  defined,  and  the 
ICOM  for  it  was  produced.  During  this  process,  no  discussion  was  allowed  regarding  how  the 
activity  would  be  implemented  in  the  prototype.  Thus  modeling  was  completed  in  the 
framework  of  how  the  CITT  could  best  achieve  a  particular  user’s  needs.  Only  after  an  activity 
had  been  designed  and  modeled  was  discussion  opened  to  implementation  in  the  prototype.  At 
that  point,  the  team  considered  the  implications  for  the  prototype  database  definitions  and  design, 
followed  by  implication  for  the  user  interface  (i.e.,  how  the  application  screens  would  function 
with  the  database),  and  by  implications  for  the  help  files. 

There  was  a  terminology  problem  that  had  to  be  overcome  before  modeling  could 
proceed  successfully.  In  the  STRUCCTT  projects,  TSP  had  been  used  to  indicate  a  set  of 
exercises  that  supported  a  specific  mission  (e.g.,  movement  to  contact,  defense  in  sector,  etc.), 
and  individual  exercises  were  referred  to  as  Tables.  In  reviewing  other  Army  training  systems, 
particularly  SATS,  it  was  apparent  that  TSP  was  used  differently.  Specifically,  the  TSP  applied 
to  an  individual  exercise.  This  discrepancy  caused  initial  difficulty  especially  for  modeling.  The 
solution  was  to  agree  on  and  document  a  consistent  terminology  wherein  TSP  was  used  to  refer 
to  the  materials  which  support  a  specific  exercise  as  was  employed  in  SATS.  What  had  been 
referred  to  as  a  TSP  in  the  STRUCCTT  projects  would  be  referred  to  as  an  exercise  mission  set. 
Once  terminological  inconsistencies  were  resolved,  modeling  proceeded  more  smoothly.  From  a 
broader  perspective,  however,  this  issue  will  require  resolution  before  integration  among  all  of 
the  various  Army  training  systems  will  be  achieved. 

Prototype  Development  and  Test 

The  decision  to  make  the  CITTSA  and  CITTDT  versions  appear  alike  created  difficulties. 
The  decision  was  based  on  a  consideration  of  appearance  versus  function  and  on  the  idea  that  by 
making  the  versions  look  alike,  users  would  have  less  difficulty  using  two  versions.  However, 
the  development  of  the  prototype  versions,  particularly  the  distributed,  was  more  difficult 
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because  of  this.  Additional  time  and  programming  were  required  to  make  the  distributed  version 
look  like  the  standalone  when  all  that  was  actually  required  was  for  the  two  versions  to  function 
the  same. 

It  is  possible  for  prototype  development  to  start  significantly  before  design  has  reached 
80%.  Although  the  Research  Plan  specified  80%,  prototype  development  actually  began  much 
earlier,  and  while  the  present  project  did  not  assess  the  impact  of  this,  it  did  demonstrate  that  it  is 
possible.  Perhaps  future  projects  can  assess  the  relative  risks  and  benefits  of  beginning  prototype 
development  at  various  stages  of  design. 

Working  with  an  off-site  developer  to  supply  a  major  piece  of  the  CITTDT  created 
difficulties.  This  was  most  likely  due  to  less  frequent  communication  between  the  on-site  and 
off-site  developers.  The  difficulties  were  not  alleviated  until  a  member  of  the  on-site  staff  was 
assigned  to  work  directly  with  the  off-site  developers,  and  a  member  of  the  off-site  developer’s 
staff  started  working  on-site  more  often.  Future  projects  that  use  off-site  personnel  to  complete 
portions  of  the  project  need  to  consider  how  to  establish  and  maintain  close  communication  and 
oversight. 

Test  results  for  the  CITTDT  were  very  dependent  on  what  time  of  day  the  test  was 
conducted  and  how  congested  the  actual  connection  from  the  Internet  Service  Provider  (ISP) 
was.  For  military  posts  the  typical  user’s  connection  is  through  the  post  LAN  with  each  user 
having  his  or  her  own  Internet  Protocol  address.  This  led  to  heavy  congestion,  particularly 
between  mid-morning  and  mid-afternoon,  and  a  degradation  of  system  performance.  For  a 
system  such  as  the  CITTDT  to  operate  efficiently,  it  needs  to  be  accessible  through  a  fast 
connection  such  as  a  proxy  server. 


SUMMARY 

This  report  has  described  the  activities  and  outcomes  of  a  project  to  develop  the 
Commanders’  Integrated  Training  Tool  for  the  Close  Combat  Tactical  Trainer.  The  CITT  will 
provide  commanders  and  other  unit  trainers  with  the  capability  to  develop  structured  training 
exercises  for  use  in  the  CCTT  virtual  trainer  including  the  ability  to  select  existing  training 
exercises  that  match  their  unit’s  needs,  and  if  no  such  exercises  exist,  to  modify  existing 
exercises  or  create  new  ones.  To  maximize  training  effectiveness,  CITT  supports  a  process  for 
exercise  modification  or  creation  in  accordance  with  the  principles  of  structured  training. 

The  primary  product  of  the  project  was  the  design  of  the  CITT  which  was  completed  and 
documented  in  accordance  with  industry  accepted  modeling  procedures  and  standards. 

Extensive  effort  was  also  directed  at  identifying  and  organizing  information  on  CCTT,  the 
structured  training  process,  and  exercise  development  which  comprised  the  10  and  was  included 
in  the  CITT  design.  The  IO  served  as  the  basis  for  the  development  of  two  videotapes  for 
CCTT — one  for  brigade  commanders  and  above,  and  one  for  brigade  commanders  and  below.  It 
also  served  as  the  basis  for  the  Learn  About  CCTT  module  of  a  CITT  prototype  which  was 
developed  as  a  proof  of  principle  of  the  CITT  design.  The  prototype  CITT  was  produced  in  two 
versions,  a  standalone  version  and  a  distributed  version,  although  the  standalone  version  had 
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greater  functionality.  Formative  evaluation  of  the  prototype  was  conducted  at  Fort  Hood  and 
Fort  Knox  using  test  cases  designed  to  exercise  all  of  the  CITT  functionalities  with  a  variety  of 
users. 


The  FE  results  were  analyzed  to  determine  defects  or  problems  with  the  CITT  and  areas 
where  CITT  could  be  improved.  Two  levels  of  results  were  obtained:  those  which  were 
implemented  in  the  CITT  during  the  project  in  order  to  produce  the  refined  CITT  prototype,  and 
those  which  were  recommended  for  fiiture  CITT  development.  Based  on  FE  and  data  captured 
during  the  project,  lessons  learned  were  derived  and  presented. 

Finally,  an  implementation  and  fielding  plan  for  the  CITT  was  developed  and  presented 
for  both  a  near-  and  long-term  solution.  The  near-term  plan  provides  for  fielding  the  CITT  as  a 
standalone  system  while  the  long-term  plan  provides  for  integrating  CITT  into  a  broader-based 
Army  Training  System  such  as  SATS. 
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APPENDIX  A 


ACRONYMS 


AAR 

After- Action  Review 

ADTDL 

Army  Doctrine  and  Training  Digital  Library 

APC 

Armored  Personnel  Carrier 

ARI 

Army  Research  Institute 

ARI AFRU 

ARI  Armored  Forces  Research  Unit 

ARTEP 

Army  Training  and  Evaluation  Plan 

ASAT 

Automated  Systems  Approach  to  Training 

AT  XXI 

Army  Training  XXI 

ATIMP 

Army  Training  Information  Management  Program 

ATSC 

Army  Training  Support  Center 

BLUFOR 

Blue  (Friendly)  Forces 

BOS 

Battlefield  Operating  System 

CATS 

Combined  Arms  Training  Strategy 

CATT 

Combined  Arms  Tactical  Trainer 

CBT 

Computer-based  Training 

CCTT 

Close  Combat  Tactical  Trainer 

CCTT-D 

Close  Combat  Tactical  Trainer  -  Digital 

CES 

Combat  Engineer  Support 

CGF 

Computer  Generated  Forces 

CIS 

Combat  Instruction  Set 

CITT 

Commanders’  Integrated  Training  Tool 

CITTDT 

CITT  Distributed 

CITTSA 

CITT  Standalone 

CLS 

Contractor  Logistics  Support 

COR 

Contracting  Officer’s  Representative 

COTS 

Commercial-off-the-shelf 

CSS 

Combat  Service  Support 

CTCP 

Combat  Trains  Command  Post 

DA 

Department  of  the  Army 

DDE 

Dynamic  Data  Exchange 

DIM 

Dismounted  Infantry  Module 

DIS 

Distributed  Interactive  Simulation 

DoD 

Department  of  Defense 

DOIM 

Directorate  of  Information  Management 

EDUCCATT 

Education  of  CCTT  through  Computer  Assisted  Training  Technology 

EIA 

Electronics  Industries  Association 

A-l 


FABTOC 

Field  Artillery  Battalion  Tactical  Operations  Center 

FDC 

Fire  Direction  Center 

FE 

Formative  Evaluation 

FEO 

For  Exposition  Only 

FIST-V 

Fire  Support  Team  Vehicle 

GCM 

Graphic  Control  Measure 

GIF 

Graphic  Interchange  Format 

GUI 

Graphical  User  Interface 

HHQ 

Higher  Headquarters 

HMMWV 

High  Mobility  Multipurpose  Wheeled  Vehicle 

HTML 

Hypertext  Markup  Language 

HumRRO 

Human  Resources  Research  Organization 

ICAM 

Integrated  Computer-Aided  Manufacturing 

ICOM 

Inputs,  Controls,  Outputs,  and  Mechanisms 

IDEF 

Integration  Definition  for  Function  Modeling 

IEEE 

Institute  of  Electrical  and  Electronics  Engineers 

10 

Instructional  Overview 

IPR 

In-process  Review 

ISD 

Instructional  Systems  Design 

ISP 

Internet  Service  Provider 

LAN 

Local  Area  Network 

MC 

Maintenance  Console 

MCC 

Master  Control  Console 

METL 

Mission  Essential  Task  List 

METT-T 

Mission,  Enemy,  Troops,  Terrain,  and  Time  Available 

MIS 

Management  Information  System 

MTP 

Mission  Training  Plan 

O/C 

Observer/Controller 

OPFOR 

Opposing  Forces 

PM 

Project  Manager 

PM  CATT 

Project  Manager  for  the  Combined  Arms  Tactical  Trainer 

POC 

Point  of  Contact 

POV 

Point  of  View 

SAF 

Semi- Automated  Forces 

SAT 

Systems  Approach  to  Training 

SATS 

Standard  Army  Training  System 

SIMNET 

Simulation  Networking 

A-2 


SME 

Subject  Matter  Expert 

SOW 

Statement  of  Work 

SQAP 

Software  Quality  Assurance  Plan 

STRICOM 

Simulation,  Training,  and  Instrumentation  Command 

STRUCCTT 

Structured  Training  for  Units  in  the  Close  Combat  Tactical  Trainer 

STX 

Situational  Training  Exercise 

SUIP 

Simple  User  Interface  Prototype 

TACP 

Tactical  Air  Control  Party 

TEXMIS 

Training  Module  (TRAMOD)  Executive  Management  Information  System 

TOC 

Tactical  Operations  Center 

TPSC 

Task  Performance  Support  Code 

TRADOC 

Training  and  Doctrine  Command 

TRAMOD 

Training  Module 

TSM 

TRADOC  System  Manager 

TSP 

Training  Support  Package 

UIP 

User  Interface  Prototype 

UISG 

User  Interface  Style  Guide 

UMCP 

Unit  Maintenance  Collection  Point 

VTP 

Virtual  Training  Program 

WOG 

Workstation  Operators  Guide 
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This  is  the  opening  screen  of  CITT.  It  provides  the  user  with  an  overview  of  the  CITT  system  and  access  to  CITT  s 
major  components. 
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The  Produce  Training  Materials  screen  provides  access  to  the  “how  to”  instruction  on  preparing  and  using  structured 
training  support  packages,  as  well  as  the  ability  to  select,  review,  or  create  a  new  exercise. 


Select  By  Task  JiEBnafflBLl 

Select  By  T  ask  I  HowTo  Select  By  Task 
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The  Select  by  Task  screen  assists  the  user  in  narrowing  down  the  selection  of  TSPs  by  entering  specific  task 
requirements.  Further,  it  provides  access  to  the  Select  by  Name  and  Select  by  Criteria  screens. 
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This  is  a  results  screen  for  the  Select  by  Task  screen.  It  displays  all  the  TSPs  meeting  the  specified  requirements. 


CITT  Exercise  Id  I  PAMV3-2CCS  ]  F  Complete 
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The  Modify  An  existing  CCTT  Exercise  screen  is  the  first  screen  in  the  modification  process.  It  outlines  all  the  steps 
involved  in  the  modification  of  an  existing  TSP. 


Select  Initial  Settings 

Name:  jDevelop  the  Situation 


The  Select  Initial  Settings  screen  allows  the  user  to  establish  the  initial  exercise  parameters.  This  screen  also  provides 
access  to  the  Matrix  OPORD  and  exercise  overlay. 
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The  Modify  Training  Objectives  screen  provides  the  user  the  ARTEP  MTP  task  that  will  be  the  focus  of  the  exercise 


t3  o 
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the  Environmental  Conditions,  and  Workstations. 
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The  Modify  Exercise  Context  screen  allows  the  user  to  Modify  the  Tactical  Situation.  This  provides  the  context  for  the 
exercise  for  the  training  unit.  Further,  it  allows  users  to  modify  the  OPFOR  SAF,  Manned  Modules,  and  BLUEFOR 
SAF  starting  vehicle  locations  and  communication  information  necessary  to  run  the  exercise. 
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The  Exercise  Event  Guide  Modification  screen  allows  the  user  to  tie  all  of  the  modification  steps  together.  This  screen 
illustrates  the  events  in  a  sequential  order  and  highlights  the  actions  the  O/C,  the  unit,  and  the  workstation  operators  are 
expected  to  execute  and  the  tasks  and  task  steps  associated  with  the  event. 


r 
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The  Workstation  Execution  Guidelines  Modification  screen  allows  the  user  to  develop  guidelines  for  each  workstation 
used  in  an  exercise. 


Complete  Modified  Exercise  TSP 
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The  Complete  Modified  Exercise  TSP  screen  provides  the  user  the  ability  to  cross-check  and  verify  all  the  changes  that 
have  been  made  to  the  exercise. 
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The  Exercise  Management  Tools  module  provides  site  personnel  access  to  key  sections  of  the  CCTT  exercise  materials. 


■xercise  Data  padfi-2CCS  Print  Materials 

System  Requirements  '  Executable  Overlay  Data  Ok  1  # 
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The  Exercise  Data  screen  provides  site  personnel  with  specific  information  about  a  given  exercise.  This  includes  the 
system  requirements,  executable  overlay  data,  and  exercise  performance  and  site  operations  notes. 
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The  Create  A  New  CCTT  Exercise  screen  is  the  first  screen  in  the  exercise  development  process.  It  outlines  all  the  steps 
involved  in  creating  an  exercise. 


[Define  Mission  Set  Parameters 
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The  Define  Mission  Set  Parameters  screen  allows  the  user  to  define  the  foundation  for  the  mission  set  and  helps  guide 
the  creation  of  the  exercise  within  the  set.  This  includes  establishing  the  initial  settings,  selecting  the  mission  set  tasks, 
determining  the  mission  set  concept,  generating  the  OPORD  materials,  and  partitioning  into  exercises. 
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identifies  the  workstations  required  for  the  exercise. 
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The  Exercise  Concept  -  Create  Exercise  TSP  screen  allows  the  user  to  convey  the  concept  of  the  exercise.  This  section 
consists  of  five  components;  the  Event  Table,  the  Exercise  Description,  the  Training  Even  Diagram,  the  Environmental 
conditions,  and  the  Workstation  Operator  Actions. 


CITT  Exercise  Id:  I  >00<XOX>00< 
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The  Complete  New  Exercise  TSP  screen  provides  the  user  the  ability  to  cross-check  and  verify  all  entries  and  make  final 
corrections  to  the  exercise. 
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COMMANDER’S  INTEGRATED  TRAINING  TOOL 
USER  SURVEY 

DATA  REQUIRED  BY  THE  PRIVACY  ACT  OF  1974 
AUTHORITY:  Title  10,  USC,  Sec  2358. 

PRINCIPAL  PURPOSE:  The  purpose  of  this  survey  is  to  collect  program  evaluation  regarding  the  Commander’s 
Integrated  Training  Tool  for  the  Close  Combat  Tactical  Trainer.  The  data  collected  with  this  form  are  to  be  used  for 
research  purposes  only. 

ROUTINE  PURPOSE:  This  as  an  experimental  data  collection  form  developed  by  the  U.S.  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences  pursuant  to  its  research  mission  as  prescribed  in  AF:  70-1. 

DISCLOSURE:  Your  participation  in  this  research  is  strictly  voluntary.  Individuals  are  encouraged  to  provide 
complete  and  accurate  information  in  the  interests  of  research,  but  there  will  be  no  effect  on  individuals  for  not 
providing  all  or  any  part  of  the  information. 

Please  complete  the  survey  each  time  you  use  CUT  even  if  you  have  completed  the  survey  previously.  To  complete 
the  survey,  click  on  “OK”. 

PT  60-18 

The  information  you  provide  by  completing  this  survey  will  be  used  to  improve  CITT  and  make  it  more  responsive 
to  the  needs  of  CITT  users.  To  complete  the  survey  type  in  your  response  in  the  area  provided  or  click  the  check 
box  for  the  desired  alternative. 

Your  current  duty  position: _ 

Is  this  your  first  time  using  the  CITT?  Yes _  No _ 

If  you  said  no,  approximately  how  many  previous  times  have  you  used  the  CITT? 


_ 1-2 

_ 3-5 

_ 6-10 

_ more  than  10 

For  what  reason  did  you  use  CITT  during  the  current  session?  Select  all  that  apply.. 

_ To  learn  about  CCTT 

_ To  learn  about  CITT 

_ To  produce  training  materials 

_ To  select  an  exercise 

_ To  modify  an  exercise 

_ To  create  an  exercise 

_ To  produce  exercise  files 

_ To  receive  system  training 


Have  you  viewed  ‘The  Senior  Leaders  Guide  to  CCTT  System  and  Training  Capabilities?”  Video 
_ Yes  _ No 
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If  yes,  how  helpful  was  the  video  to  your  use  of  the  CITT? 

_ Extremely  helpful 

_ Very  helpful 

_ Somewhat  helpful 

_ Slightly  helpful 

_ Not  at  all  helpful 

Have  you  viewed  ‘The  Unit  Leaders  Guide  to  Training  in  the  CCTT”  Video?  _ Yes  - No 

If  yes,  how  helpful  was  the  video  to  your  use  of  the  CITT? 

_ Extremely  helpful 

_ Very  helpful 

_ Somewhat  helpful 

_ Slightly  helpful 

_ Not  at  all  helpful 

How  would  you  describe  your  level  of  experience  with  using  personal  computers? 

_ Very  experienced 

_ Moderately  experienced 

_  Slightly  experienced 

_ Complete  novice 

What  specifically  did  you  want  to  accomplish  using  the  CITT  during  this  session?  (For  example,  “Modify  exercise 
PAD  IF,  Create  a  new  exercise,  etc.”) 


Using  the  CITT,  I  was  able  to  accomplish  what  I  wanted  to  do. 

_ Strongly  agree 

_ Agree 

_ Neither  agree  nor  disagree 

_ Disagree 

_ Strongly  disagree 

How  difficult  do  you  think  it  is  to  navigate  through  the  CITT? 

_ Extremely  easy 

_ Very  easy 

_ Easy 

_ _  Difficult 

_ Very  difficult 

_ Extremely  difficult 

Navigation  tools  and  buttons  are  used  consistently  throughout  the  CITT. 

_ Strongly  agree 

_ Agree 

_ Neither  agree  nor  disagree 

_ Disagree 

_ Strongly  disagree 
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If  I  became  “lost” ’  in  the  CUT,  it  was  usually  easy  to  get  back  to  where  I  wanted  to  be. 


_ Strongly  agree 

_ Agree 

_ Neither  agree  nor  disagree 

_ Disagree 

_ Strongly  disagree 

Did  you  use  the  Help  feature  of  the  CITT  during  this  session?  Yes _  No - 

If  yes,  approximately  how  many  times? 

_ 1-2 

_ 3-5 

_ 6-10 

_ 10-20 

_ more  than  20 

The  built-in  CITT  “Help”  Material  is  very  helpful. 

_ Strongly  agree 

_ Agree 

_ Neither  agree  nor  disagree 

_ Disagree 

_ Strongly  disagree 

The  “Help”  material  is  written  at  the  correct  level  for  my  needs. 

_ Strongly  agree 

_ Agree 

_ Neither  agree  nor  disagree 

_ Disagree 

_ Strongly  disagree 

Was  there  any  time  during  this  session  when  you  needed  “Help”  but  were  unable  to  obtain  it?  Yes. 
No _ 

If  Yes,  please  explain: _ _  ... - 

What  feature  of  the  CITT  did  you  find  the  most  useful? 


What  feature  of  the  CITT  did  you  find  least  useful? 


If  you  could  add  a  feature  to  the  CITT,  what  would  it  be? 


Additional  comments: 
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FORMATIVE  EVALUATION  DATA  COLLECTION  FORMS 


CITT  Test  Procedure 


Note:  Each  CUT  test  will  be  observed  by  two  observers. 

1 .  One  of  the  two  observers  will  instruct  the  subject  using  the  appropriate  Test  Session 
Introduction. 

2.  Following  the  instructions,  the  subject  will  be  provided  with  the  Test  Case  for  the  session. 

3.  During  the  test,  each  observer  will  record  his  observations  in  the  CITT  Test  Observation 
Booklet  in  accordance  with  the  Instructions  contained  in  the  booklet. 

4.  Following  completion  of  the  Test  Case,  the  subject  will  complete  the  CITT  on-line  survey 
(for  each  test  case  completed) 

5.  Following  completion  of  all  test  cases  for  an  individual  subject,  the  observers  will  conduct  a 
Debriefing  Session  in  accordance  with  the  instructions  contained  in  the  CITT  Test  Debriefing 
Booklet. 
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DIRECTIONS  FOR  USE: 


Procedure 

Place  your  name  on  the  front  of  the  booklet.  Each  FE  observer  will  be  provided  with  his/her  individual 
copy. 

At  the  beginning  of  each  test  session,  complete  the  Test  Case  Identification  information.  Record  the 
Subject’s  name,  the  date,  the  test  case  identification,  and  the  start  time  of  the  test. 

During  the  test  session,  record  your  observations  on  the  portion  of  the  sheet  labeled  “Observations.”  Each 
time  the  subject  accesses  “Help”,  put  a  tally  in  the  “Help”  column  at  the  point  in  your  observation  at 
which  Help  was  accessed.  At  the  completion  of  the  test,  record  the  Stop  Time  and  whether  the  test  was 
completed  successfully.  Use  as  many  recording  sheets  as  required  for  the  session. 

Rationale 

In  accordance  with  the  Formative  Evaluation  plan,  we  arc  particularly  interested  in  the  time  taken  to 
complete  each  module,  the  paths  the  user  takes  through  CITT,  whether  a  successful  outcome  was  reached, 
the  types  of  navigation  problems  that  occurred,  any  system  errors  that  occurred,  and  difficulties  the  user 
experienced  as  he/she  used  the  CITT.  (Other  FE  data  will  be  collected  through  surveys  and  interviews.) 
The  goal  is  to  capture  as  much  of  the  session  as  practical,  however,  based  on  prior  user  testing,  we  need  to 
recognize  that  we  can  not  record  every  action  taken  by  the  subject  nor  every  question/comment  made.  To 
facilitate  recording,  opposite  each  recording  sheet  is  an  Observation  Guidance  form  which  contains  a 
shorthand  process  for  recording  most  CITT  actions.  For  example,  if  the  subject  is  completing  the  action 
Create  -  Context  -  Starting  Location,  you  would  record  this  as  CX2. 

Providing  Assistance 

At  the  bottom  of  the  Observation  Guidance  form  is  a  box  labeled  Exercise  Assistance.  This  shows  the 
progression  of  assistance  the  subject  should  be  provided  in  the  event  he/she  experiences  difficulties  with 
the  test.  Our  overall  goal  is  passive  observation,  however,  based  on  prior  user  testing,  it  is  anticipated  that 
assistance  will  occasionally  be  required. 

When  required,  observers  should  provide  the  minimum  assistance  necessary.  The  first  action  should  be  to 
suggest  that  the  subject  try  the  CITT  Help.  This  will  often  be  sufficient  to  solve  the  problem.  If  that  is 
not  successful,  provide  a  general  hint  or  assistance.  For  example,  “you  might  want  to  consider  the  task 
steps  you  want  the  unit  to  perform.”  If  that  is  still  not  sufficient,  provide  a  direct  instruction.  For 
example,  “Go  to  Select  Initial  Settings  and  list  the  task  steps  you  want  the  unit  to  perform.” 

As  assistance  is  provided  during  a  session,  it  is  important  that  the  details  be  captured  and  recorded.  We 
need  to  note  when  assistance  was  required,  the  nature  of  the  assistance  provided,  and  the  extent  of  the 
assistance  provided.  The  need  for  assistance  may  represent  a  flaw  or  weakness  in  CITT,  however,  we 
will  not  be  able  to  determine  this  unless  we  have  detailed  information  on  the  assistance  provided. 
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OBSERVATION  GUIDANCE 


Use  the  following  abbreviations  for  designating  CITT  actions  performed  by  the  user 


Navigate  by  Role 

Q)  UnitCDR/Ldr 

(2)  CCTT  Site  Staff 

(3)  USW  Operator 

(4)  O/C 


by  Need 

(1)  Learn  about  CCTT 

(2)  Review  Ex’s 

(3)  Select  Ex’s 

(4)  Modify  Ex’s 

(5)  Create  Ex’s 


(6)  Plan  and  Build  Ex’s 

(7)  Plan  for  Training 

(8)  Prepare  for  Training 

(9)  Learn  h/t  Exec.  Training 

(10)  Learn  h/t  Assess  Tmg 


Learn  About  CITT 

Select  an  Exercise 

by  Task 

by  Name 

by  Criteria 

Review  an  Exercise 
Summary 

Outline  TSP 

Matrix  OPORD  Overlay 

Plan  Sheets 

THumbnail 

Mission 

Modify  Ex 

Select  Init  Settings 

Concept 

ConteXt 

Event  Guide 

CD  Tasks 

CD  Ex.  Desc. 

(D  Tact.  Sit. 

CD  O/C  Action 

(2)  Task  Steps 

(2)  TED 

(2)  Starting  Loc 

(2)  Unit  Action 

Steps 

Actions 

(3)  Events 

(4)  Event  Actions 

(5)  Event  Tasks 

Wkstat.  Exec.  Gd 

(3)  Environment 

(4)  Workstations 

Complete  Mod.  TSP 

Q)  Commo  Data 

(3)  Task/Task 

(4)  Wkstat. 

(D  Plan  Sheets 
(2)  Ex.  TSP 


Create  Ex  Mission  Set  Parms 

(1)  Init  Settings 

(2)  Mission  Set  Tasks 

(3)  Mission  Set  Concept 

(4)  OPORD  Mtls 

(5)  Partition  Set 


Exercise  Outlines 

(1)  Ex.  Description 

(2)  TED 

(3)  Tactical  Situation 

(4)  OPFOR  &  BLUFOR  Sit 

(5)  Ex.  Tasks  and  Task  Steps 

(6)  Designate  Workstations 


Concept 
CD  Ex.  Desc. 

(2)  TED 

(3)  Environment 

(4)  Workstations 


ConteXt 

(1)  Tact.  Sit. 

(2)  Starting  Loc 

(3)  Commo  Data 


Event  Guide 

(1)  O/C  Action 

(2)  Unit  Action 

(3)  Task/Task  Steps 

(4)  Wkstat.  Actions 


Wkstat.  Exec.  Gd 


Complete  Mod.  TSP 
(D  Plan  Sheets 
(2)  Ex.  TSP 


Produce  Exercise  Files 


Exercise  Management  Tools 


Test  Assistance: 

1.  Maintain  passive  observation 

2.  Suggest  user  review  help 

3.  Provide  suggestions  (e.g.,  general  information  on  CITT) 

4.  Provide  specific  activity  to  execute 
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S: _ 

Date: _ 

Test  Case: 


of 


Sheet. 
Start  Time:. 
Stop  Time: 


S  successfully  completed  test  case:  Yes _  No. 
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Guidelines  for  Debriefing  Session  following  CITT  Test 

The  debriefing  should  occur  after  the  subject  has  completed  the  on-line  CITT  survey. 

The  debriefing  supplements  the  data  collected  during  observation  and  from  the  survey  and  can 
be  used  to  elaborate  on  data  previously  collected.  The  debriefing  should  be  conducted  using  a 
conversational  format,  however,  the  general  categories  listed  below  should  be  covered. 

1.  Follow  up  on  any  items/concems  remaining  from  the  CITT  test  observations,  for  example, 
any  questions  you  made  note  of  during  the  test  re:  why  a  subject  took  a  particular  action. 

2.  Ask  subject  for  his  overall  impression  of  CITT  in  terms  of: 

•  Look  and  feel 

•  Ease  of  navigation 

•  Ease  of  use 

3.  Ask  subject  about  the  information  included  in  Learn  About  CITT: 

•  Value  of  the  content 

•  Readability 

•  Use  of  audio  and  video 

4.  Ask  subject  about  the  on-line  help. 

•  Relevance  to  reason  he  accessed  help 

•  Satisfactorily  answered  his  questions 

5.  Ask  subject  what,  if  anything  in  CITT,  made  it  difficult  to  complete  his  task. 

6.  Ask  subject  what  features  of  CITT  were  especially  useful. 

7.  Ask  subject  what,  if  anything,  he  would  like  to  have  had  in  CITT  that  wasn’t  included. 

8.  Ask  subject  for  any  other  comments  he  wishes  to  provide. 
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CITT  Test  Subject  Debriefing 


Session 
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DIRECTIONS  FOR  USE: 


Procedure 

This  booklet  contains  all  of  the  recording  forms  for  use  in  the  Build/Proof  phase  of  the  CITT  formative 
evaluation  at  Fort  Hood.  Place  your  name  on  the  front  of  the  booklet. 

The  booklet  contains  three  types  of  forms:  recording  forms  for  Build/Proof  observations, 
interview/debriefing  recording  forms  for  CLS  personnel,  and  interview/debriefing  recording  forms  for 
Unit  personnel. 

Build/Proof  Observations 

During  the  Build/Proof  activities,  record  your  observations  on  the  forms  provided.  In  accordance  with  the 
Formative  Evaluation  plan,  we  are  interested  in  whether  the  exercises  selected,  modified,  or  created  using 
CITT  support  the  Build/Proof  process.  It  is  not  feasible  or  necessary  to  provide  a  contemporaneous 
record  of  the  Build/Proof  process;  rather,  we  are  interested  in  actions  and  outcomes  which  were 
particularly  effective  or  particularly  ineffective  to  the  Build/Proof  process.  The  recording  forms  provide 
space  for  the  following:  a  description  of  the  action/outcome;  whether  the  action/outcome  was  effective  or 
ineffective;  and  your  analysis  of  the  factors  (CITT  and  non-CITT)  which  contributed  to  the 
effectiveness/ineffective-ness  of  the  action/outcome. 

CLS  Personnel  Interview/Debrief 

The  CLS  Personnel  Interview/Debrief  recording  forms  contain  questions  to  be  used  during  the  debriefing 
of  the  site  CLS  personnel  who  were  involved  in  the  Build/Proof  activities.  The  interview/debriefing  will 
probably  require  30-60  minutes  and  should  be  conducted  as  a  group. 

Unit  Personnel  Interview/Debrief 

The  Unit  Personnel  Interview/Debrief  recording  forms  contain  questions  to  be  used  during  the  debriefing 
of  the  site  CLS  personnel  who  were  involved  in  the  Build/Proof  activities.  The  interview/debriefing  will 
probably  require  30-60  minutes  and  should  be  conducted  as  a  group. 
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Observation  and  Interview/Debriefing  Guidelines  for  Build/Proof  test 

The  original  conception  for  this  portion  of  FE  was  that  we  would  observe,  in  total,  the  site 
personnel  building  the  exercises  modified  or  created  by  the  unit,  and  that  we  would  observe  the 
site  personnel  and  unit  personnel  proofing  the  exercises.  It  is  quite  possible  that  the  work  the  site 
will  have  completed  prior  to  the  build/proof  test  will  completely  confound  the  results  obtained. 
The  exercises  will  likely  have  been  built  prior  to  the  unit’s  use  of  CITT.  If  this  is  the  case,  we 
will  need  to  decide  whether,  or  to  what  extent,  data  collected  during  this  week  are  usable. 

Ideally,  we  will  collect  the  following  three  types  of  data: 

1 .  Observation  of  the  Build/Proof  process.  This  observation  will  be  conducted  using  a 
significant  incident  approach.  That  is,  the  observer  will  look  for  and  record  actions/events 
which  occur  that  are  not  part  of  a  standard  build/proof  process;  rather,  they  represent  things 
that  worked  particularly  well  or  particularly  poorly.  The  observer  will  not  attempt  to  conduct 
contemporaneous  observation.  Observations  will  be  recorded  in  the  Build/Proof  Data 
Recording  Booklet. 

2.  Interview/debriefing  data  from  CCTT  CLS  personnel  involved  in  the  Build/Proof  test. 
Interview  notes  will  be  recorded  in  the  Build/Proof  Data  Recording  Booklet. 

3.  Interview/debriefing  data  from  Unit  personnel  involved  in  the  Build/Proof  test.  Interview 
notes  will  be  recorded  in  the  Build/Proof  Data  Recording  Booklet. 
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Build/Proof  Observation  Form 


Action/Outcome: 


_ Effective  _ Ineffective 

Contributing  factors: _ 


_ Effective  _ Ineffective 

Contributing  factors: _ 


Guidelines  for  Unit  Personnel  Debriefing  Session  following  Build/Proof  Test 

The  debriefing  should  occur  after  the  Build/Proof  of  all  exercises  have  been  completed. 

The  debriefing  supplements  the  data  collected  during  the  Build/Proof  observation.  The 
debriefing  should  be  conducted  as  a  group  activity  using  a  conversational  format,  however,  the 
items  listed  below  should  be  covered. 

1 .  Follow  up  on  any  items/concems  remaining  from  the  Build/Proof  observations. 

2.  Ask  the  following: 

•  Overall  impression  of  the  Build/Proof  process  using  exercises  produced  with  CUT. 

•  Was  this  your  first  experience  with  Building/Proofing  CCTT  exercises? 

•  Were  all  necessary  materials  for  the  Build/Proof  produced  by  CITT? 

•  What  materials  were  missing? 

•  Did  the  CITT  provide  you  with  the  information  you  needed  to  participate  effectively  in  the 
Build/Proof  process? 

•  Did  the  exercise  materials  produced  by  CITT  support  the  Build/Proof  process? 
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Unit  Personnel  Interview/Debriefing 
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Guidelines  for  CLS  Personnel  Debriefing  Session  following  Build/Proof  Test 

The  debriefing  should  occur  after  the  Build/Proof  of  all  exercises  have  been  completed. 

The  debriefing  supplements  the  data  collected  during  the  Build/Proof  observation.  The 
debriefing  should  be  conducted  as  a  group  activity  using  a  conversational  format,  however,  the 
items  listed  below  should  be  covered. 

1 .  Follow  up  on  any  items/concems  remaining  from  the  Build/Proof  observations. 

2.  Ask  the  following: 

•  Overall  impression  of  the  Build/Proof  process  using  exercises  produced  with  COT. 

•  Were  all  necessary  materials  for  the  Build/Proof  produced  by  COT? 

•  What  materials  were  missing? 

•  Were  the  materials  provided  in  a  format  that  supported  the  Build/Proof  process? 

•  Were  there  any  problems  with  Building/Proofing  the  exercises?  If  so,  what  were  they? 

•  What,  if  any,  relationship  to  these  problems  have  to  CITT? 

•  Was  the  Build/Proof  process  easier,  more  difficult,  or  about  the  same  for  exercises  produced 
using  the  COT  compared  to  other  exercises  you  have  done? 
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CLS  Personnel  Interview/Debriefing 
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Observation  and  Interview/Debriefing  Guidelines  for  Exercise  Execution 

The  purpose  of  this  part  of  the  COT  Formative  Evaluation  is  to  determine  through  observation 
and  interviews  whether  or  not  the  unit  was  able  to  successfully  execute  the  exercises  that  had 
been  developed  using  CITT.  We  will  collect  the  following  types  of  data: 

1.  Observation  of  the  Exercise  Execution  process.  The  [Formative  Evaluation}  Test  plan  for 
COT  specifies  that  we  will  collect  data  on  difficulties  encountered/errors  that  occurred 
during  Exercise  Execution.  We  will  combine  observation  with  a  modified  significant 
incident  reporting  technique  to  accomplish  this.  That  is,  the  observer(s)  will  look  for  and 
record  events  that  occurred  and  that  indicate  difficulties/errors  with  the  execution  process. 
The  observer  will  not  attempt  to  conduct  contemporaneous  observation.  Observations  will 
be  recorded  in  the  Exercise  Execution  Data  Recording  Booklet 

2.  Interview/debriefing  data  from  COT  CLS  and  Unit  personnel  involved  in  the  Exercise 
Execution  test.  Interview  notes  will  be  recorded  in  the  Exercise  Execution  Data  Recording 
Booklet. 
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DIRECTIONS  FOR  USE: 


Procedure 

This  booklet  contains  the  recording  forms  for  use  in  the  Exercise  Execution  phase  of  the  CITT  formative 
evaluation  at  Fort  Hood.  Place  your  name  on  the  front  of  the  booklet. 

The  booklet  contains  two  types  of  forms:  recording  forms  for  Exercise  Execution  observations,  and 
interview/debriefing  recording  forms  for  CLS  and  Unit  personnel. 

Exercise  Execution  Observations 

During  the  Exercise  Execution  activities,  record  your  observations  on  the  forms  provided.  In  accordance 
with  the  Formative  Evaluation  plan,  we  are  interested  in  whether  the  Unit  was  able  to  successfully 
execute  the  exercises  selected,  modified,  or  created  using  CITT.  It  is  not  feasible  or  necessary  to  provide 
a  contemporaneous  record  of  the  Exercise  Execution  process;  rather,  we  are  interested  in  actions  and 
outcomes  which  indicate  difficulties  or  errors  in  the  execution  of  the  exercises.  The  recording  forms 
provide  space  for  the  following:  a  description  of  the  error/difficulty,  and  your  analysis  of  the  factors 
(CITT  and  non-CITT)  which  contributed  to  the  difficulty/error. 

CLS  and  Unit  Personnel  Interview/Debrief 

The  CLS  and  Unit  Personnel  Interview/Debrief  recording  forms  contain  questions  to  be  used  during  the 
debriefing  of  the  site  CLS  and  Unit  personnel  who  were  involved  in  the  Exercise  Execution  activities. 
The  interview/debriefing  will  probably  require  30-60  minutes  and  should  be  conducted  as  a  group. 
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Guidelines  for  CLS  and  Unit  Personnel  Debriefing  Session  following  Exercise  Execution 

The  debriefing  should  occur  after  the  Exercise  Execution  of  all  exercises  have  been  completed. 

The  debriefing  supplements  the  data  collected  during  the  Exercise  Execution  observation.  The 

debriefing  should  be  conducted  as  a  group  activity  using  a  conversational  format,  however,  the 

items  listed  below  should  be  covered.  Note:  a  given  item  may  be  more  relevant  to  one  group  of 

participants  than  to  others. 

1.  Follow  up  each  difficulty/error  recorded  from  the  Exercise  Execution  observations. 
Determine  what  the  CLS  and  Unit  personnel  thought  went  wrong  and  whether  the  problem 
can  be  related  to  CUT  (i.e.,  would  the  difficulty/error  have  occurred  in  any  event  or  did  it 
occur  as  a  result  of  CUT  having  been  used  to  develop  the  exercise  and/or  train  the  Unit 
personnel)? 

2.  Ask  the  following: 

•  For  each  Unit  Support  Workstation  Operator,  did  the  training  they  received  in  CITT 
adequately  prepare  them  to  support  the  exercises?  If  not,  ask  for  specific  examples  of  lack  of 
preparation  or  lack  of  required  knowledge? 

•  Did  the  exercises  as  executed  adequately  match  the  exercises  as  “intended”  when  they  were 
developed  in  CITT?  If  not,  ask  for  specific  instances  or  examples  of  a  mismatch.  How  are 
the  mismatches  related  to  the  CITT  (e.g.,  CITT  provided  incorrect  information,  insufficient 
information,  etc.)? 

•  What,  if  any,  features/functions  of  the  CITT  were  particularly  helpful  in  the  execution  of  the 
exercises? 

•  What,  if  any,  features/functions  of  the  CITT  created  particular  difficulties  in  the  execution  of 
the  exercises? 

•  Having  gone  through  exercise  selection/modification/creation,  build/proof,  and  execution, 
what  is  their  overall  impression  of  CITT? 

•  What  would  they  most  like  to  see  added  to  CITT? 

•  What  could  be  removed  from  CITT? 
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CLS  and  Unit  Personnel  Interview/Debriefing 
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APPENDIX  F 


CUT  IMPROVEMENTS  IMPLEMENTED 

These  are  refinements  that  can  be  accomplished  within  the  CITT  project  time  frame.  They  are 

based  on  findings  from  formative  evaluation. 

1 .  Help  needs  to  link  to  the  FM’ s. 

2.  In  Help,  the  contents  tab  in  the  treeview  frame  is  broken.  It  does  not  display  contents  - 
instead  it  links  to  “How  to  select  an  Exercise”. 

3.  There  are  other  links  that  are  broken. 

4.  The  OPORD  link  was  broken  -  test  participant’s  OPORD  was  saved,  but  was  not  associated 
with  the  exercise  he  was  creating. 

5.  In  the  10,  make  the  bullets  hyperlinks.  Users  click  them  expecting  them  to  link  to  the  topic. 

6.  In  the  10,  Forward  and  Back  still  do  not  always  work  as  expected. 

7.  The  blank  Matrix  OPORD  in  the  CREATE  has  some  formatting  applied  to  the  fields  (e.g., 
Some  are  bold,  etc.)  Clean  up  the  formatting. 

8.  Fill  in  all  “Note”  fields  for  existing  exercises. 

9.  For  screens  that  use  tabs  (Workstation  Execution  Guidelines,  etc.),  make  the  tabs  stand  out. 
Many  test  subjects  missed  the  tabs. 

10.  System  Requirements  Table  is  not  accurate. 

1 1 .  The  10  index  is  missing  a  lot  of  key  words.  Redo  the  index. 

12.  In  10,  change  “Click  <  to  view”  to  “Click  the  <  button  above  to  view.” 

13.  Eliminate  the  screen  numbers  from  the  title  bar. 

14.  Make  the  exit  button  from  10  more  obvious. 

15.  Simplify  the  select  screen.  Perhaps  give  the  select  options  on  the  first  screen,  then  have 
following  screens. 

16.  On  the  Plan  Sheet,  under  Marksmanship,  replace  current  values  with  those  used  in  CCTT. 

17.  In  Review  Exercise  TSP,  the  Titles  of  all  four  sections  should  be  included  at  the  top  of  the 
screen  with  the  selected  one  grayed  out. 
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18.  Disable  Print  from  the  toolbar. 


19.  Change  “Done”  to  “Completed”  on  the  Select  Screen. 

20.  In  OPORD,  if  you  have  multiple  annexes,  make  them  links. 

21.  In  10,  fix  the  picture  of  the  guy  who’s  going  to  get  his  elbow  cut  off  (it  is  in  Characteristics 
of  Structured  Training). 

22.  In  Create  Mission  Set,  the  user  can  inadvertently  create  a  single  exercise  mission  set  (one  of 
the  test  participants  did)  and  there’s  no  way  to  correct  it  A  couple  of  possibilities:  have  the 
default  value  for  the  partition  box  be  0  so  that  user  has  to  enter  something.  Or  ask  for  a 
confirmation  when  they  select  the  number  of  exercises  in  the  set. 

23.  In  Workstation  Actions,  expand  the  text  box.  Users  thought  they  only  had  one  line  to  enter 
text. 

24.  Delete  the  drop  down  list  for  Unit  ID  in  Plan  Sheets. 

25.  In  10,  the  Treeview  frame  should  always  come  up  in  Contents  mode  the  first  time  the  user 
accesses  10. 

26.  Correct  “sheath”  to  “sheaf’  in  10  where  it  talks  about  FABTOC/AFATDS. 

27.  Change  "The  commander  enters  fire  support  data  ..."  to  read  "The  FDO  enters  fire  support 
data. . . This  is  in  the  FABTOC/AFATDS  portion  of  the  IO. 

28.  Check  to  make  sure  we  always  use  AFATDS,  not  AFT  ADS.  (In  the  10  part  on 
FABTOC/AFATDS). 
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APPENDIX  G 


CITT  IMPROVEMENTS  TO  BE  IMPLEMENTED 


These  are  based  on  formative  evaluation  and  other  design  considerations.  They  represent 

modifications  and  enhancements  that  can  not  be  implemented  in  the  prototype  CUT. 

1 .  Allow  user  to  select  multiple  tasks  or  task  steps  simultaneously  versus  having  to  select  them 
one  at  a  time  and  going  back  to  the  drop  down  list  each  time. 

2.  Provide  the  capability  for  the  user  to  review  TSP  materials  by  role. 

3.  Provide  the  capability  for  users  to  print  TSP  materials  by  role.  This  will  allow  users  to  print 
only  the  portions  of  the  TSP  they  need  to  execute  exercises. 

4.  Provide  the  capability  for  users  to  print  selected  portions  of  the  TSP  to  include  individual 
pages. 

5.  Redesign  the  way  users  add  or  delete  units  in  exercises.  Make  the  activity  more  closely 
resemble  either  a  task  organization  chart  or  an  MCC  like  screen.  Have  this  activity 
automatically  build  plan  sheets  and  the  exercise  initialization  file  for  the  exercise. 

6.  Provide  a  simple  activity  for  the  user  to  add  or  modify  “relocatables”  such  as  obstacles  and 
fighting  positions.  Integrate  this  activity  with  the  map  tools  and  have  it  automatically 
populate  the  plan  sheets  and  the  exercise  initialization  file  for  the  exercise. 

7.  Provide  the  capability  to  create  the  TED  by  cropping  the  overlay  and  map  created  for  the 
exercise  set. 

8.  Provide  the  capability  to  associate  existing  overlay  files  with  new  exercises. 

9.  Make  the  create  activity  a  multi-session  activity.  . 

10.  Provide  an  option  for  modifying  exercise  sets  vice  just  individual  exercises. 

1 1.  Provide  the  capability  to  update  Training  Event  Diagrams  when  modifying  the  operations 
overlay. 

12.  In  Plan  Sheets  and  Starting  locations  -  do  not  list  the  Ml A2  as  an  option;  provide  look-up 
tables  in  future  CITT. 

13.  Allow  the  user  to  copy  an  exercise  to  a  floppy  so  he  can  take  to  another  machine. 

14.  Add  scroll  bars  where  full  text  is  not  displayed,  even  when  that  field  is  not  the  focus. 
Requires  Visual  Basic. 
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15.  Redesign  Plan  Sheets  based  on  task  organizing.  This  will  require  adding  task  guidance. 
Include  mobility/survivability. 

16.  Re-engineer  Create  and  Modify  to  be  more  user  oriented,  i.e.,  use  military  terms  and  make  it 
more  like  METT-T. 

17.  Add  tutorials  for  Create  and  Modify  that  actually  complete  an  exercise  creation  or 
modification  as  the  user  completes  the  tutorial. 

18.  Design/redesign  CUT  for  users  other  than  commanders. 

19.  Provide  an  “undo”  button. 

20.  Do  not  allow  the  user  to  exit  Access  using  the  X  button.  Provide  an  intercept  message  asking 
for  confirmation. 

21.  Redesign  the  Select  screen  (which  actually  involves  redesign  of  the  selection  process.) 

22.  Redesign  the  “triptik”  that  is  generated  in  the  Navigate  screen.  Include  links  to  the 
appropriate  content  and  activities.  Customize  for  the  various  users  (i.e.,  do  not  just  have  a 
“Workstation  operator”  option,  but  make  it  the  “FABTOC  Workstation  operator.” 

23.  Include  overlays  for  all  exercises. 

24.  Provide  for  force  on  force  exercises. 

25.  Examine  and  revise  the  Event  Guide  to  simplify. 

26.  All  Object  Linking  and  Embedding  objects  (training  event  diagrams,  Matrix  Opord,  Overlay, 
Commo  Matrix,  SOI  Extract)  need  to  be  broken-down  into  data  elements  and  normalized  into 
the  table  structure.  CITT  2  should  not  contain  any  embedded  objects  or  hyperlinks  to  external 
programs. 

27.  Lookup  tables  need  to  be  developed  and  populated  for  the  Plan  Sheet  fields:  System,  Unit  ID, 
Unit  Type  and  Equipment  ID.  Each  table  needs  to  have  restrictive  criteria  identified  and 
specific  fields  created. 

28.  TRADOC  Educational  System  code  for  interpreting  map  location  of  graphic  elements  needs 
to  build  a  link  between  the  Training  Event  Diagram  and  the  Plan  Sheets,  and  if  possible, 
Training  Event  Diagram  graphics  would  be  constructed  from  data  stored  in  tblPlanSystem 
(also  possibly  Overlays — but  this  would  need  additional  research  owing  to  the  structural 
differences  as  the  Plan  Sheets  relate  to  the  exercise  and  the  Overlay  relates  to  the  Set). 

29.  Access  2000  needs  to  be  assessed  for  both  front  end  and  back  end  capabilities  before 
development  is  moved  to  SQL  and  Visual  Basic. 
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30.  (If front-end  is  VB)  All  CITTSA  forms  need  to  be  redesigned  in  Visual  Basic  6.0.  Subforms 
must  be  designed  unique  to  one  parent  form  to  eliminate  the  problem  of  topic-specific  help 
with  subforms.  Forms  that  show  the  event  guide  and  the  workstation  operator  guidelines 
should  be  reconstructed  with  the  assumption  that  VB  will  allow  more  than  two  embedded 
subforms.  The  form  to  add  actions  for  an  event  would  not  be  necessary.  The  Plan  Sheets 
need  to  be  redesigned  to  accommodate  removing  the  ammunition  fields  from  tblPlanSystem 
and  normalizing  them  into  one  field  in  a  child  table. 

31.  (If  front-end  is  Access  2000)  Queries  that  were  developed  just  to  provide  a  sort  order  to  data 
displayed  on  forms  need  to  be  deleted,  forms  sourced  to  a  table  and  the  Form_Load  event 
needs  to  include  “OrderByOn  =  True.” 

32.  (If front-end  is  Access  2000)  Remove  the  close,  min  and  max  buttons  from  forms  (except  for 
dialogs). 

33.  Access  Replication  technology  (or  the  SQL  equivalent)  needs  to  be  implemented  to  allow 
exercises  created  on  one  copy  of  C1TT  to  be  integrated  into  the  central  database. 
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